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1.0  INTRODUCTION 


The  Anticipating  Job  Aiding  and  Training  Requirements  research  effort  was  sponsored 
by  the  Air  Force  Research  Laboratory’s  Logistics  Readiness  Branch  (AFRL/RHAL)  under  Task 
Order  #9  of  the  Technology  for  Agile  Combat  Support  (TACS)  contract  (FA8650-04-D-6546). 
The  period  of  performance  for  the  research  effort  extended  from  13  July  2007  to  12  January 
2009.  This  report  documents  the  technical  effort  and  results  of  research  activities  conducted  as 
part  of  this  task  order.  The  scope  of  these  activities  included  the  following: 

■  Reviewing  relevant  literature  related  review  of  current  and  future  trends  impacting 
the  workforce,  work  environment,  and  technologies  relevant  to  job  aiding  and 
training. 

■  Gathering  information  through  telephone  interviews  with  representatives  of  Air  Force 
organizations  involved  in  force  development  and  training,  as  well  as  Air  Force  Career 
Field  Managers  responsible  for  establishing  training  requirements. 

■  Planning  and  conducting  workshops  focused  on  gaining  a  better  understanding  of  the 
Air  Force  Technical  Training  environment,  challenges,  and  future  directions. 

■  Identifying  and  formulating  potential  future  research  vectors  intended  to  better  inform 
future  programs,  policies,  and  decisions  impacting  Air  Force  Technical  Training. 

■  Conducting  a  pilot  study  of  21st  Century  learning  and  teaching  styles  in  select  courses 
conducted  at  the  82nd  Training  Wing  (82nd  TRW),  Sheppard  AFB,  TX. 

1.1  Objective  and  Scope 

The  primary  objective  of  this  research  effort  was  to  gain  a  better  understanding  of  current 
trends  (demographic,  workforce,  technology,  etc.)  and  challenges  related  to  job  aiding  and 
training  in  order  to  develop  the  foresight  needed  to  help  formulate  potential  areas  for  future 
research.  During  the  course  of  this  effort,  the  scope  of  our  research  and  investigations  became 
specifically  focused  on  Air  Force  technical  training,  which  encompasses  training  for  career  fields 
such  as  Aircraft  Maintenance,  Medical,  Civil  Engineering,  and  numerous  others  conducted  at 
various  Air  Force  bases  (as  well  as  sister  service  bases)  in  the  U.S.  Although  not  specifically 
focused  on  during  the  research  in  the  context  of  technical  training,  the  topic  of  job  aiding  was 
still  considered  relevant  to  our  research,  since  some  of  the  same  job  aids  used  to  directly  support 
day-to-day  operations  and  maintenance  activities  in  the  field,  are  also  an  integral  part  of 
technical  training.  For  example,  Interactive  Electronic  Technical  Manuals  (IETMs)  and  Core 
Automated  Maintenance  Systems  (CAMS)  workstations  are  used  to  support  certain  course 
blocks/units  of  instruction  in  aircraft  maintenance  training  courses.  From  this  perspective,  the 
job  aids  used  are  more  properly  classified  as  training  equipment  which  was  a  consideration  in  the 
research. 

1.2  Background 


The  Air  Force  faces  many  challenges  in  the  future  due  in  part  to  the  changing  nature  of 
warfare  (e.g.  cyberspace),  a  changing  workforce  (e.g.  loss  of  skilled  workers)  and  an  increasing 
operations  tempo  -  fueled  in  part  by  the  current  War  on  Terror.  Addressing  these  challenges  puts 
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a  strain  on  personnel  and  other  resources,  and  adversely  impacts  budget  outlays  in  numerous 
areas  including  system  procurement,  equipment  refurbishment,  research  and  development,  and 
numerous  others  including  technical  training.  With  respect  to  education  and  training,  the  Air 
Force  needs  to  develop  a  common  vision  for  the  future  (Air  Education  and  Training  Command 
(AETC)  Future  Learning  Board  presentation,  Sep  08)  to  help  address  some  of  these  challenges, 
including  devising  a  creative  plan  and  systematic  approach  for  adapting  new  strategies  and 
methods  for  developing  and  conducting  technical  training,  as  well  as  for  integrating  or  inserting 
new  technologies  that  can  help  meet  training  objectives  in  a  more  efficient  and  cost  effective 
manner. 


2.0  STUDY  APPROACH  AND  METHODOLOGY 

The  core  research  team  for  this  effort  was  comprised  of  personnel  (contractor  and  AFRL) 
representing  several  academic  disciplines  including  operations  research,  human  factors, 
education  and  training  (including  previous  AF  instructor  experience),  and  computer  science. 

The  composition  of  the  team  allowed  us  to  take  a  multi-discipline  approach  to  the  task  of 
assessing  and  formulating  potential  research  areas  related  to  job  aiding,  education  and  training. 
The  model  or  framework  for  conducting  the  research  is  portrayed  in  Figure  1.  The  research 
conducted  as  part  of  this  effort  commenced  with  a  literature  search  and  review  of  relevant 
publications  and  technical  reports  on  factors  (e.g.  demographics,  technology,  etc.)  impacting 
training,  and  in  particular  Air  Force  technical  training.  The  intent  of  this  investigation  was  to 
identify  current  and  future  trends  in  relevant  areas  and  assess  their  importance  to  formulating 
future  research  vectors  for  job  aiding  and  technical  training.  Interviews  and  discussions  were 
conducted  with  personnel  at  HQ  USAF,  HQ  AETC,  2nd  Air  Force,  and  the  82nd  TRW  at 
Sheppard  AFB,  TX  to  gain  a  better  understanding  of  current  policies  and  processes  related  to  the 
selection,  classification,  and  training  of  enlisted  personnel,  and  future  challenges  in  these  areas. 


•  Workforce 

•  Demographics 

•  Cultural  /  Social  Factors 

•  Learning  Styles 

•Training  and  Education 

•  Training  Content  /  Methods 

•  Delivery  Mechanisms 

•  Work  Environment 

•  Work  Practices 

•  Operational  Requirements 

•  Future  Systems  /  Needs 

•  Technology 

•  Job  Performance  Aiding 

•  Educational  /  Training 


B  Research  areas  should 
evolve  from  a  human- 
centered,  integrated 
analysis 

encompassing  all 
areas 


Assess  Current 
States  and  Future 
Trends 


Identify 
jv  Potential 
^  Research 
Areas  (Gaps) 


Figure  1:  TACS  DO-9  Research  Framework 
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In  addition  to  these  activities,  two  workshops  were  planned  and  convened  as  part  of  the 
research  effort.  The  first  workshop  was  hosted  and  supported  by  the  82nd  TRW  at  Sheppard 
AFB,  TX  in  January  2008.  The  primary  intent  of  this  workshop  was  to  gain  a  better 
understanding  of  the  problem  space  as  it  pertained  to  the  challenges  associated  with  training 
future  warfighters.  A  second  workshop  was  convened  at  Wright  State  University  in  April  2008. 
This  workshop  brought  together  members  of  the  core  research  team  to  discuss  and  identify 
potential  areas  and  topics  that  future  research  efforts  might  focus  on  to  help  address  challenges 
identified  through  the  literature  search  activities  and  interviews  with  Air  Force  personnel,  as  well 
as  observations  and  technical  discussions  at  the  initial  workshop. 

The  final  portion  of  the  research  effort  focused  on  planning  and  conducting  a  pilot  study 
to  assess  the  learning  and  teaching  styles  of  students  and  instructors,  as  well  as  the  instructional 
methods  used,  in  a  select  group  of  courses  conducted  at  Sheppard  AFB.  The  methodology, 
analysis,  and  results  of  this  pilot  study  are  documented  in  a  separate  report  included  in  Appendix 
D  and  considered  in  the  development  of  recommendations  documented  in  this  report. 

3.0  RESULTS  AND  DISCUSSION 

3.1  Literature  Review  and  Interviews 

3.1.1  Relevant  Factors  and  Demographic  Shifts 

Three  key  factors  are  projected  to  have  important  effects  on  the  workforce  and  work 
environment  in  the  next  10  to  15  years.  These  factors  include  demographic  shifts,  technological 
advances,  and  global  competition  [Ref.  1],  [Ref.  2].  In  turn,  these  factors  (collectively  in  some 
cases)  are  causing  the  skill  and  knowledge  demands  related  to  work  to  increase,  and  thus 
spurring  the  call  for  more  rapid  improvements  in  both  primary  and  secondary  education  (grades 
K-12  and  beyond),  particularly  in  Science,  Technology,  Engineering,  and  Mathematics  (STEM) 
curriculums.  With  respect  to  demographic  shifts,  current  and  future  trends  indicate  that  the 
growth  rate  of  labor  force  is  expected  to  slow  to  0.6  percent  in  the  2010s  and  0.4  percent  in  the 
2020s  [Ref.  3],  [Ref.  4].  That  is  a  sharp  decline  from  the  1.3  percent  average  annual  increases 
seen  in  the  1990s  and  the  2.6  percent  average  annual  increases  experienced  during  the  1970s.  At 
the  same  time,  the  workforce  grows  older.  In  was  estimated  that  over  50  percent  of  the 
workforce  in  2007  was  eligible  to  retire  [Ref.  5],  thus  raising  concerns  about  how  to:  a)  replace 
or  offset  the  loss  of  job  knowledge  and  experience  of  those  retiring,  and  b)  retraining  those  in 
this  category  that  elect  to  continue  to  work,  but  will  need  to  learn  and  develop  new  skills  in 
response  to  shifting  job  demands  caused  by  a  global  economy  and  technological  advancements. 
The  workforce  of  the  future  is  also  expected  to  be  more  racially/ethnically  diverse.  One 
important  question  that  comes  into  play  as  it  pertains  to  Air  Force  recruitment  and  technical 
training  is  given  these  shifts,  how  do  we  change  the  way  we  recruit  and  train  future  warfighters? 

3.1.2  The  “Millennial”  Generation 

The  American  youth  population,  ages  16  to  21,  is  the  primary  recruiting  market  of  the 
Armed  Services,  and  their  characteristics  have  been  changing  over  time.  Gradual  changes  are 
occurring  in  the  racial/ethnic  composition  of  this  population,  as  well  as  their  educational  and 
career  aspirations.  Almost  all  youth  (88  percent)  report  they  plan  to  continue  schooling  and 
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obtain  higher  education  of  one  form  or  another.  The  number  of  high  school  graduates  and  the 
number  of  youth  completing  some  college  is  increasing  steadily.  However,  as  the  number  of  high 
school  graduates  going  on  to  attend  college  increases,  the  cost  of  a  post- secondary  education  has 
also  increased  dramatically  [Ref.  6] . 

The  youth  comprising  this  population  are  part  of  the  “Millennial”  generation  -  a 
generation  that  includes  over  75  million  Americans  born  between  1982  and  2000  (circa).  This 
means  current  and  future  (at  least  for  several  years)  recruitment  and  training  efforts  in  the  Air 
Force  (as  well  as  sister  services)  will  be  targeted  towards  individuals  representing  the  Millenial 
generation.  Some  key  characteristics  of  Millenials  include  the  following: 

■  Technologically  savvy 

■  View  technology  as  permeating  every  aspect  of  their  lives 

■  Racially  and  ethnically  diverse 

■  Identify  with  their  parents  values  and  feel  close  to  their  parents 

■  Rely  heavily  on  collaboration  with  peers  as  a  form  for  learning 

■  Learning  needs  to  be  hands-on,  interactive,  collaborative,  and  fun 

■  Need  for  on-going  learning 

The  learning  preferences  (or  learning  style)  of  Millenials  favor  teamwork,  experiential 
activities,  structure,  and  technology  and  their  learning  strengths  include  multi-tasking,  goal 
orientation,  a  collaborative  style,  and  positive  attitudes  [Ref.  7].  In  addition,  there  is  mounting 
evidence  that  Millenials  have  a  different  relationship  with  information  and  learning  than  those 
from  previous  generations,  primarily  due  to  their  access  to  the  Internet  and  computer-enabled 
technologies  [Ref.  8].  Arguably,  one  potential  drawback  of  this  access  to  technology  is  that 
Millenials,  by  age  21,  tend  to  spend  significant  less  time  reading  -  less  than  5,000  hours  -  and 
more  time  with  activities  such  as  playing  video  games  or  talking  on  a  cell  phone  -  equally 
divided,  around  20,000  hours  [Ref.  9] . 

From  a  training  perspective,  the  characteristics,  learning  preferences,  and  technology 
driven  traits  of  Millennial  cited  above  should  be  important  considerations  in  the  design, 
development  and  delivery  of  technical  training,  including  the  insertion  of  training  technologies 
into  course  curriculum.  However,  to  our  knowledge,  no  direct  consideration  or  recognition  to 
individual  learning  styles  is  given  in  course  development  processes  such  as  the  Instructional 
System  Design  (ISD)  process  that  has  long  been  used  by  the  Air  Force  to  develop  training 
courses. 

3.1.3  Technical  Training  Challenges 

The  Air  Force  continues  to  face  many  training  challenges  due  to  a)  an  increased 
operational  tempo  associated  with  supporting  on-going  contingency  and  wartime  operations 
abroad;  b)  budgetary  and  force  reduction  pressures;  and  c)  transformation  initiatives  such  as  Air 
Force  Smart  Operations  21st  Century  (AFS021),  Maintenance  Training  Enterprise  2010,  and 
Virtual  Training  Record  that  evolved  from  the  eLog-21  program.  Some  of  the  key  challenges  for 
Air  Force  training  include  the  following: 


4 


■  Reduced  budgets  for  providing  training,  including  manpower,  training  equipment, 
and  other  resources 

■  Reduction  and  consolidation  of  Air  Force  specialties 

■  Increased  pressure  on  mid-level  and  senior  enlisted  supervisors  to  fill  the  “voids”  in 
technical  training  that  cannot  be  addressed  in  initial  skills  training  at  AETC  formal 
training  schools.  The  increased  demands  associated  with  supporting  operations  in 
Iraq  and  Afghanistan  afford  less  time  for  training  in  the  field 

■  Increased  need  to  reduce  training  time  and  get  personnel  to  the  field  with  the  initial 
skills  they  need  to  perform  effectively  in  their  respective  jobs 

■  Loss  of  skilled  personnel  due  to  an  aging  workforce  and  personnel  retiring  or  exiting 
the  Air  Force 

■  Reducing  course  “washback”  (repeating  blocks  of  instruction)  and  attrition  rates  in 
select  courses 

In  reference  to  the  last  bullet  concerning  course  attrition  and  washback  rates,  it  is  worth 
noting  for  this  research  that  a  snapshot  assessment  of  these  rates  in  the  last  calendar  year 
quarter  of  2007  indicated  that  for  the  82nd  TRG  at  Sheppard  AFB,  TX,  32  percent  of  initial 
skills  courses  reported  “high”  (above  programmed  goals)  attrition,  and  approximately  50 
percent  of  courses  had  washback  rates  exceeding  16  percent.  These  factors  are  significant, 
considering  that  the  average  cost  for  initial  skills  training  in  the  Air  Force  exceeded  $47,000 
per  trainee  in  FY  07,  with  training  for  some  career  fields  like  crypto  linguists  exceeding 
$100,000  (Source:  AFI  65-503,  Table  A18).  Figure  2  identifies  technical  training  courses  in 
2nd  AF  (including  Sheppard  AFB)  that  had  high  attrition  or  washback  rates  that  exceeded  20 
percent  in  FY07  (only  courses  with  50  or  more  students  were  considered  in  the  analysis).  The 
rates  shown  were  compiled  from  data  in  the  Technical  Training  Management  System 
(TTMS)  for  FY07.  A  subset  of  these  courses  was  investigated  in  Air  Force  sponsored 
RAND  study  that  will  be  addressed  later  in  this  report. 
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ELIMINTATIONS 

GOODFELLOW 

WASHBACKS 

GOODFELLOW 

COURSE 

TITLE 

50+ COUNT 

20%  +  RATE 

COURSE 

TITLE 

50  +  COUNT 

20%+  RATE 

XBAQZ1 N335A0ADA 

Arabic  Basic 

52 

288.89% 

XBABR1N531  0A1A 

Electronic  Signals  Intelligence  Exploitation 

56 

81.16% 

X3ABR1N031  0A6A 

Operations  Intel  Apprentice 

113 

21.52% 

X5AQZ1 A831 KP040 

Airborne  Linguist,  Korean  Basic  (Disc) 

73 

24.50% 

XBOVZXXXX  0ADA 

Arabic  Basic 

78 

96.30% 

X5AQZ1N334G  041 

Far  East  Crypto  Linguist  Apprentice,  Korean  (Disc) 

203 

27.88% 

XBAQZ1 A831 D0KP  A 

Airborne  Korean  Basic 

224 

29.83% 

XBAQZ1 N335A0ADA 

Arabic  Basic 

497 

24.58% 

ELIMINTATIONS 

KEESLER 

WASHBACKS 

KEESLER 

COURSE 

TITLE 

50  +  COUNT 

20%  +  RATE 

COURSE 

TITLE 

50  +  COUNT 

20%+  RATE 

E3ABR1C131  00RB 

Air  Traffic  Control  Radar  Apprentice 

54 

21.77% 

E3ABR1C131  00TB 

Air  Traffic  Control  Tower  Apprentice 

52 

25.00% 

E8ABR1W031  1A1B 

Aerographer's  Mate  Class  A1 

61 

29.90% 

E3AQR2E633  0E2B 

EP,  Telephone  &  Switching  Systems 

63 

22.34% 

E3AQR2A533A048B 

EP,  Communications/Navigation/Mission  Systems 

112 

21.62% 

E8ABR1W031  0A1A 

Weather  Forecaster  Apprentice 

121 

31.35% 

E3ABR1C1 31  00RB 

Air  Traffic  Control  Radar  Apprentice 

148 

37.37% 

E3ABR2E231  01ZC 

Network  Infrastructure  System 

188 

30.23% 

ELIMINTATIONS 

LACKLAND 

WASHBACKS 

LACKLAND 

COURSE 

TITLE 

50  +  COUNT 

20%  +  RATE 

COURSE 

TITLE 

50  +  COUNT 

20%  +  RATE 

LCAQP1XXXX 01 OA 

Air  Force  Combat  Dive  Course  -  Open  Circuit 

54 

37.50% 

L3ABR2T031  00AA 

Traffic  Management  Apprentice 

81 

19.90% 

L3AQR1C231  0C0B 

Combat  Control  (Prelim) 

101 

45.29% 

L3ABP1C431  0T0A 

Tactical  Air  Command  &  Control  Apprentice 

115 

30.10% 

L3ABP1C431  0T0A 

Tactical  Air  Command  &  Control  Apprentice 

108 

40.45% 

L3AQR1T031  0S0A 

Survival  Evasion  Resistance  Escape  (Prelim) 

137 

52.09% 

L3AQR3E831  0E0B 

Explosive  Ordnance  Disposal  (Prelim) 

148 

52.11% 

L3AQR1T231  0P1A 

Pararescue  Indoctrination  Course 

163 

57.60% 

ELIMINTATIONS 

SHEPPARD 

WASHBACKS 

SHEPPARD 

COURSE 

TITLE 

50  +  COUNT 

20%  +  RATE 

COURSE 

TITLE 

50  +  COUNT 

20%  +  RATE 

JBAQN3E831  00NA 

Explosive  Ordnance  Apprentice 

50 

38.17% 

J8ABR4A231  00AA 

DoD  Biomedical  Maintenace  Equip  Technician  (AF) 

53 

43.09% 

J8ABR4A231  20AA 

DoD  Biomedical  Maintenance  Equip  Tech 

57 

50.89% 

J3ABR2A631 E050B 

Aerospace  Propul  Appr  Jet  Eng  (F1 1 0)  (MRT) 

56 

23.83% 

J3ABR2A631 C048B 

Aerospace  Propul  Appr,  Jet  Eng  (TF33,  TF34,  F-108) 

72 

21.88% 

J8ABR3E131  00AB 

HVAC  &  Refrigeration  Apprentice 

76 

24.68% 

J3ABR2A636  0A9B 

Aircraft  Elect  &  Environmental  Systems  Apprentice 

122 

19.65% 

JBAQN3E831  00NA 

Explosive  Ordnance  Apprentice 

128 

49.42% 

J8ABR4A231  20AA 

DoD  BiomedMaint  Equipment  Technician  (Army) 

221 

65.00% 

J3ABR2A632  046B 

Aerospace  Ground  Equipment  Apprentice  (AGE) 

257 

30.96% 

J3AQR4N031  01AB 

Aerospace  Medical  Service  Apprentice  -  Phase  1 

1,093 

44.27% 

Includes  performance,  academic  and  medical  washbacks  and  eliminations 

Figure  2:  2nd  AF  Washbacks/Eliminations  (FY07) 


The  Air  Force  is  investigating  ways  to  streamline  training  and  address  some  of  the 
challenges  outlined  above.  For  example,  the  Maintenance  Training  Enterprise  Value  Stream 
Mapping  project,  started  in  early  2006,  is  an  AFSO-21  initiative  intended  to  help  streamline  the 
Air  Force  technical  training  process  by  leaning  out  inefficiencies  and  redundancies  in  the  process 
and  viewing  training  from  a  “life-cycle”  perspective.  Based  on  discussions  with  a  few  Air  Force 
Career  Field  Managers,  two  of  the  more  significant  questions  that  might  be  addressed  in  future 
research  include  1)  how  can  we  best  leverage  technology  to  innovate  and  streamline  training?; 
and  2)  how  do  we  address  the  impact  of  an  aging  workforce  and  loss  of  skilled  technicians  to 
train  future  warfighters?  There  was  also  some  recognition  of  the  need  to  better  understand  the 
learning  needs  and  styles  of  future  warfighters  (Millenials),  to  better  understand  how  we  can  best 
leverage  technology,  and  adapt  selection  and  placement  processes/tools,  to  meet  their  learning 
needs  as  well  as  the  needs  of  the  Air  Force. 

3.1.4  Future  of  Air  Force  Education  and  Training 

One  of  the  more  significant  papers  reviewed  included  a  HQ  AETC  White  Paper  released 
31  Jan  2008  titled  On  Learning:  The  Future  of  Air  Force  Education  and  Training  [Ref.  10].  The 
paper  is  intended  to  articulate  concepts  the  AF  should  consider  adopting  to  train  and  educate 
future  Airmen  -  the  Millennial  or  21st  Century  Learners.  These  concepts  include  Knowledge 
Management,  Continuous  Learning,  and  Precision  Learning.  Some  of  the  key  assumptions  that 
underpin  the  concepts  and  recommendations  presented  in  the  plan  include  the  following: 

■  The  Air  Lorce  will  innovate  and  change  its  approach  to  education  and  training  in 
order  to  effectively  prepare  future  Airmen  to  perform  successfully. 
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■  New  approaches  to  education  and  training  will  account  for  how  future  Airmen  learn, 
as  well  as  changes  in  technology,  and  the  future  environment  in  which  Airmen 
operate. 

■  Future  enterprise-wide  learning  systems  will  access  comprehensive,  authoritative  data 
and  information  and  will  be  widely  accessible,  highly  reliable,  and  robustly  protected. 

■  An  integrated  and  systematic  approach  to  improving  the  quality  and  delivery  of  the 
learning  experience  will  focus  and  leverage  current  and  future  education  and  training 
initiatives. 

■  To  lead-turn  adversary  learning,  the  Air  Force  will  be  an  early  tester  and  adopter  of 
new,  innovative  learning  approaches  and  technologies  or  risk  falling  behind. 

■  Knowledge-enabled  Airmen  will  continue  to  be  the  key  to  flexibility  and  the  Air 
Force’s  asymmetric  military  advantage  in  air,  space  and  cyberspace. 

Numerous  recommendations  are  proposed  and  discussed  in  the  paper  to  help  achieve  the 
proposed  to  realize  the  vision  of  “delivering  unrivaled  air,  space,  and  cyberspace  education  and 
training.”  One  of  the  key  recommendations,  relevant  to  this  research  task,  is  the  development  of 
MyBase,  a  virtual  environment  (world)  based  on  research  and  software  developed  by  San 
Francisco  based  Linden  Research  and  launched  in  1993.  MyBase  is  currently  a  set  of  regions  or 
islands  in  Second  Life  devoted  to  sharing  Air  Force  history  and  learning  about  the  Air  Force, 
however  current  interest  and  endeavors  are  focused  on  evolving  MyBase  as  an  environment  and 
foundation  to  support  future  Air  Force  education  and  training  initiatives  (with  limits).  As  of 
Spring  08,  efforts  were  focused  Marketing  of  white  paper  and  drafting  implementation  plan  to 
support  vision  and  recommendations.  Some  of  the  key  points  of  interest  for  the  DO-9  research 
effort  expressed  directly  or  indirectly  in  the  paper  included: 

■  “Precision  identification  of  viable  recruits  using  advanced  aptitude  and  skill 
assessment  tools.”  This  implies  new  or  better  methods  and  tools  beyond  the  currently 
administered  Armed  Services  Vocational  and  Aptitude  Battery  (ASVAB)  test  that  is 
used  to  inform  career  field  placement  decisions 

■  Too  much  batch  processing  -  need  new  concepts  that  recognize  people  learn 
differently  and  at  different  paces 

■  “Delivery  of  training  must  be  flexible  and  must  permit  schedule,  delivery  and  media 
formats  tailored  to  individual  needs” 

■  “Determine  which  technologies  are  essential”  to  the  concepts  and  “learning  vision” 
put  forth  in  the  white  paper 

■  “Learning  experiments”  that  encompass  “tests  of  various  combinations  of  virtual, 
live,  constructive  training  approaches  to  evaluate  optimal  mixes  of  technology, 
organizations,  and  learning  concepts” 

■  Paper  advocates  “a  program  for  learning  research  and  development,  and  insertion  of 
technologies  into  training  and  education  programs” 

3.2  Workshop  #1 

The  initial  DO-9  workshop  was  convened  at  the  82nd  TRW,  Sheppard  AFB,  TX  from  30 
January  2008  to  31  January  2008.  This  purpose  of  this  workshop  was  twofold:  1)  to  gain  a  better 
understanding  of  the  technical  training  environment  at  Sheppard  AFB;  2)  to  identify  additional 
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questions  we  still  need  to  address  and  information  we  need  to  acquire  to  help  better  formulate 
and  articulate  future  research  needs  as  it  pertains  to  technical  training.  The  format  and  specific 
topics  covered  at  the  workshop  are  covered  in  detail  in  Appendix  A  and  included  multiple 
subject  area/topic  presentations  (e.g.  field  evaluation  and  training  process),  course  observations, 
a  group  interview  and  panel  discussion  with  instructors,  and  DO-9  working  group  discussions  on 
information  collected  at  the  workshop.  This  initial  workshop  was  of  tremendous  value  to  the 
DO-9  task  in  helping  to  familiarize  research  team  members  with  the  overall  technical  training 
process  and  environment,  as  well  as  providing  insight  to  some  of  the  challenges  associated  with 
training  future  warfighters.  Of  particular  interest  was  the  feedback  received  from  instructors 
who  participated  in  the  panel  discussion  at  the  workshop.  A  short  questionnaire  was  developed  to 
help  facilitate  this  discussion  (see  Appendix  B).  Areas  of  primary  interest  highlighted  by 
instructors  during  the  panel  discussion  included  the  following: 

■  “Hands-on”  training  critical  to  some  courses  (e.g.  maintenance)  this  is  what  gets  the 
students  excited 

■  Quality  of  Trainees  (skills  and  motivation  to  study  /  learn  lacking  for  some  students) 

■  Remedial  Training  Needed  (in  some  courses  -  e.g.  handtools) 

■  Instructor  additional  duties  /  taskings  can  be  a  distraction  to  training 

■  See  cycles  of  “good”  students  (some  college)  and  poorer  students 

■  Some  language  barrier  problems 

■  Instructor  manning  levels  -  not  always  optimal,  and  additional  duties  for  instructors 
(and  trainees)  outside  of  class  can  be  personnel  “stressors” 

■  Funding  for  training  equipment  /  trainers  /  etc. 

■  Pressures  to  reduce  training  time  and  get  proficient  airmen  to  the  field  quicker 

In  addition  to  the  panel  discussion,  the  instructor  questionnaire  (Appendix  B)  was  also 
posted  on  the  82nd  TRW  Community  of  Practice  so  that  instructors  not  participating  in  the 
workshop  could  have  an  opportunity  to  respond.  A  total  of  60  responses  were  received  from 
instructors  representing  multiple  courses  conducted  at  Sheppard  AFB.  The  range  of  instructor 
experience  extended  from  six  months  to  13  years,  with  an  average  instructor  experience  level  of 
around  three  years.  A  summary  of  the  responses  to  each  of  these  questions  is  provided  in 
Appendix  C.  Overall,  the  responses  received  from  this  group  of  instructors  confirmed  some  of 
the  same  themes  and  challenges  cited  above  from  the  panel  discussion,  in  particular,  the 
importance  of  “hands-on”  training,  shortage  of  instructors  and  impact  of  instructor  additional 
duties  on  workload,  as  well  as  the  condition  of  training  equipment  and  funding  for  equipment 
maintenance  and  procurement  of  new  equipment. 

Members  of  the  research  team  convened  a  separate  working  group  session  at  the 
workshop  to  discuss  the  information  presented  at  the  workshop,  including  general  impressions 
from  classroom  observations  in  select  courses.  The  intent  of  this  session  was  to  try  and  identify 
some  key  needs  or  areas  of  focus  that  should  be  considered  in  the  process  of  formulating  future 
research  vectors  as  part  of  the  second  workshop.  Some  of  the  ideas  and  general  impressions 
discussed  during  this  session  included  the  following: 

■  Examining  current  processes  and  procedures  for  training  instructors  in  terms  of  the 
time  required  to  produce  qualified  instructors  which  can  now  take  up  to  nine  months. 
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■  Developing  better  policy  and  guidelines,  and  collect  data  (once  defined),  to  help  us 
downstream  better  understand  the  effective  and  efficient  use  of  technology  -  where  is 
it  warranted,  what  type  of  technology,  and  the  return  on  investment. 

■  Conducting  experiments  to  better  understand  where  technology  works  and  doesn’t 
work  in  terms  of  improving  training  and  learning  outcomes. 

■  Gaining  a  better  understanding  of  who  our  “learners”  are  and  how  they  learn. 

■  Investigating  better  methods/tools  to  assess  and  "pre-train"  for  the  requisite 
skills/knowledge  in  the  assigned  career  field. 

■  Identifying  the  relationship  between  training  “attributes”  and  the  acquisition  of 
training  technologies  -  there  seems  to  be  very  little  data  /  research  on  this  topic. 

■  Gaining  a  better  understanding  of  how  students  succeed  (or  fail)  in  the  training  that  is 
currently  provided.  For  those  that  are  not  so  successful,  what  can  be  provided  to  help 
them  have  a  better  chance  to  succeed? 

■  There  seems  to  be  a  need  to  collect  more  relevant  course  data  on  students  who 
graduate  and  those  who  do  not  due  to  academic  or  other  reasons  (attrition).  What 
were  their  profiles  like?  Are  there  leverage  points  that  we  might  learn  from  and 
exploit  to  design  future  training  or  bring  to  bear  in  the  near  term? 

■  Some  concern  about  how  much  it  appeared  policies  and  procedures  seemed  to  stifle 
(impede)  innovation  as  it  applies  to  improving  training.  Also,  the  mental  mindsets 
that  seemed  to  be  formed  around  (influenced  by)  the  organizational  culture. 

■  A  need  to  focus  on  “output  quality”  as  it  relates  to  training,  which  may  not 
necessarily  be  tied  to  training  budgets/funding  as  it  appears  now.  This  relationship  is 
important  to  understand  and  should  be  actually  scrutinized  by  the  leadership.  For 
instance,  if  the  quality  of  airmen  graduating  improved,  would  it  offset  the  emphasis 
on  funding?  Main  point  is  quality  versus  funding  -  we  need  to  better  understand  the 
relationship. 

■  Gain  a  better  understanding  course  attrition  and  washback  rates  in  terms  of  root 
causes.  Note:  During  the  timeframe  of  the  DO-9  research  effort,  a  HQ  USAF/A1PF 
sponsored  RAND  study  titled  “Enhanced  Testing  and  Screening”  was  ongoing  that, 
in  part,  was  focused  on  looking  at  schoolhouse  attrition  in  select  courses.  Data 
collected  as  part  of  that  study  may  be  helpful  for  providing  more  insight  into  root 
causes  for  attrition/washback  rates. 

3.3  Workshop  #2 

The  information  gathered  during  the  initial  workshop,  as  well  as  the  ideas  generated  and 
general  impressions  discussed  in  the  working  group  session  were  carried  forward  for 
consideration  in  a  second  workshop  convened  at  Wright  State  University  in  late  April  2008  and 
included  members  of  the  DO-9  research  team  (AFRL  and  contractor).  The  team  reviewed  the 
information  collected  in  the  first  workshop  as  well  as  additional  information  and  data  collected 
after  the  first  workshop,  and  focused  on  developing  a  framework  for  identifying  and  discussing 
potential  areas  for  future  research  as  it  pertains  to  technical  training.  It  was  clear  from  the 
discussions  and  findings  at  the  first  workshop  that  some  of  the  training  challenges  and  needs 
identified  are  significantly  impacted  by  factors  that  will  most  likely  not  be  resolved  or  addressed 
through  research  alone.  Budgetary  constraints  (e.g.  replacing  or  upgrading  training  equipment, 
trainers),  current  operational  tempo  (e.g.  pressure  to  increase  throughput  of  students  to  support 
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on-going  operations),  and  current  organizational  and  training  policies  and  procedures,  are  factors 
that  also  need  to  be  considered  as  part  of  formulating  a  future  research  agenda  for  technical 
training. 

With  this  in  mind,  members  of  the  research  team  proceeded  to  develop  a  framework  for 
discussion  of  research  that  attempted  to  address  the  various  components  or  facets  of  training  that 
were  subsequently  labeled  as  training  dimensions  (see  Fig.  3).  The  training  dimensions 
identified  include:  methodology  which  deals  primarily  with  instructional  methods,  but  also  the 
design  and  development  of  course  curriculum;  technology  which  encompasses  the  software, 
hardware,  and  infrastructure  used  to  develop  and  deliver  training;  learning  environment  which 
considers  classroom  layout  and  environmental  factors,  the  structure  of  the  day  in  terms  of 
classroom  and  outside  activities/duties,  etc.;  the  learner,  including  their  learning 
preferences/styles,  abilities,  motivation,  etc.;  the  instructor ,  including  the  quality  of  their 
training,  expertise,  attitudes  toward  use  of  technology,  etc;  and  finally,  evaluation  which 
encompasses  the  policies,  methods,  tools,  and  metrics  used  in  assessing  training,  including 
learner  and  instructor  performance,  training  methods,  and  training  technologies.  It  is  important 
to  note  a)  that  these  training  dimensions  are  not  independent,  and  in  fact  any  one  dimension  may 
have  multiple  points  of  intersection  with  others  in  future  research;  and  b)  collectively,  these 
dimensions  should  all  be  focused  on  supporting  the  goal  of  learning.  The  framework  depicted  in 
Figure  3  is  representative  of  the  perspective  the  DO-9  research  team  took  to  further  identify 
specific  attributes  for  each  training  dimension  that  were  considered  important  to  developing 
research  “vectors”  for  each  training  dimension.  The  discussion  of  each  of  these  vectors  that 
follows  is  not  intended  to  identify  specific  research  tasks  or  products,  but  rather  focuses  on 
highlighting  potential  questions  and  issues  that  should  be  considered  in  formulating  a  research 
agenda  for  technical  training. 
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Figure  3:  Training  Dimensions  Perspective  and  Attributes 


3.3.1  Learners 

This  research  vector  focuses  on  understanding  how  applying  knowledge  about  learners, 
both  individually  and  collectively,  could  improve  the  efficiency  and  success  of  technical  training. 
Trainees’  experiences  and  needs  as  learners  must  be  understood  and  optimized  in  the  technical 
training  environment.  The  “Human  Behavior  in  Military  Contexts  ”  book  (section  dealing  with 
technology  and  training  and  discussed  earlier)  concluded  that  first  and  foremost  “training 
programs  should  be  based  on  an  understanding  of  how  people  leam  and  how  instructional 
methods  affect  learning.”  [Ref.  11] 

Research  is  needed  to  better  understand  the  role  of  learners’  needs  and  individual 
differences;  most  specifically  those  attributes,  learning  preferences  (discussed  in  the  pilot  study 
conducted  as  part  of  the  DO-9  research  effort  and  included  in  Appendix  D),  pre-requisite 
knowledge  and  skills  that  contribute  to  success  or  failure  during  technical  training.  Such 
knowledge  about  learners  could  then  inform  the  redesign  of  technical  training  so  as  to  improve 
learner  engagement,  comprehension,  and  successful  completion  of  training.  This  research  has  the 
potential  to  yield  both  short  and  long  term  impact  by  identifying  areas  for  immediate  intervention 
via  training  modifications  or  redesign,  as  well  as  identifying  areas  where  developing  future 
technologies  may  improve  outcomes. 

This  research  vector  would  explore  technical  training  through  the  lens  of  learners’  needs 
and  experiences.  This  dimension  of  inquiry  will  focus  on  understanding  the  relationships 
between  learners’  physical,  psychological,  social,  and  cognitive  needs,  and  the  structure  and 
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context  of  training.  A  specific  subset  of  this  research  will  investigate  the  impact  of  interactions 
between  individual  learners’  attributes,  pre-requisite  knowledge  and  skills,  preferred  learning 
modalities,  and  instructional  methods  on  training  performance.  The  learner  vector  of  research  has 
the  potential  to  uncover  leverage  points  of  intervention  to  reduce  training  time  while  improving 
the  quality  of  learning  and  graduation  rates. 

Research  in  psychology,  biology,  linguistics,  and  neurophysiology  inform  our 
understanding  of  the  human  brain  and  learning,  and  provide  a  foundation  for  exploring  how 
learning  in  technical  training  environments  can  be  improved.  The  following  examples  are 
provided  to  illustrate  research  opportunities  in  this  vector. 

■  Current  research  on  the  plasticity  of  the  human  brain  suggests  that  learning  impacts 
brain  chemistry,  reinforcing  neural  connections  through  practice.  Cognitive  functions 
involve  the  simultaneous  interactions  of  whole  concepts  and  whole  images, 
similarities,  differences,  and  relationships.  The  brain  works  by  analogy  and  metaphor, 
and  does  not  assemble  thoughts  by  bits  of  data.  How  might  technical  training  be 
redesigned  to  optimize  these  ways  in  which  the  brain  works? 

■  Human  learning  psychology  research  suggests  that  metacognition,  thinking  about 
your  thinking,  is  a  vital  component  of  comprehension,  self-regulation,  and 
motivation.  How  might  metacognition  be  incorporated  into  technical  training  to 
optimize  motivation  and  learning? 

■  New  research  on  multi-tasking  suggests  that  learning  is  impaired  by  activities  that 
distract  a  learner’s  attention  from  the  primary  focus  of  comprehending  new 
information.  How  might  this  understanding  inform  the  selection  of  instructional 
methodologies  in  the  technical  training  environment? 

■  Recent  research  on  the  relationship  between  exercise  and  cognition  suggests  that 
movement  of  our  muscles  produces  proteins  that  play  a  pivotal  role  in  higher  level 
thinking  and  learning.  How  might  this  understanding  of  the  relationship  between 
exercise  and  brain  function  influence  the  way  we  schedule  the  training  day? 

■  Human  learning  is  an  essentially  social  endeavor.  Relationships  with  peers  and 
instructors  contribute  to  or  distract  from  learning.  How  might  this  understanding  of 
the  role  relationships  play  in  learning  guide  training  methodology? 

■  Every  learner  brings  their  unique  personality  and  life  experiences  to  the  training 
context.  Individual  attributes,  such  as;  aptitudes,  personality  characteristics,  learning 
preferences,  and  predispositions,  contribute  to  success  or  failure  in  training.  If  we 
better  understood  the  role  individual  attributes  play  in  a  learner’s  success,  how  might 
training  be  redesigned  to  better  meet  individual  trainees’  learning  needs? 

■  Over  40  years  ago,  Vygotsky  said  “culture  is  cognition,”  suggesting  that  how  we 
think  is  based  upon  our  culture  and  our  language.  The  culture  of  the  21st  Century  is 
impacting  language  and  cognition.  The  necessary  components  of  learning  in  the  21st 
century  are  not  those  focused  on  in  the  past.  Digital  age  literacy,  inventive  thinking, 
effective  communication,  and  high  productivity  must  be  included  in  future  learning 
context.  How  might  technical  training  be  redesigned  to  optimize  this  shift  in  culture 
and  cognition? 

■  Human  learning  is  essentially  constructive;  learning  builds  on  a  learner’s  prior 
knowledge  and  skills.  When  the  necessary  prerequisite  knowledge  and  skills  are  not 
in  place,  learning  is  sabotaged.  How  might  training  be  more  efficiently  accomplished 
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is  each  trainee  was  equipped  with  the  necessary  prerequisite  knowledge  and  skills, 
through  screening  processes  and  or  remediation? 

From  these  examples,  it  is  clear  that  the  “Learner”  vector  offers  a  rich  potential  research 
agenda,  and  it  intersects  with  the  other  potential  vectors  described  in  this  report.  Learners  are  the 
sine  qua  non  of  learning,  and  every  training  endeavor  must  begin  and  end  with  meeting  learners’ 
needs  if  technical  training  is  to  be  efficient  and  effective. 

3.3.2  Instructors 

This  research  vector  focuses  on  understanding  the  role  of  the  instructor,  instructor 
preparation,  instructor  professional  development,  and  instructional  modifications  that  could 
improve  the  efficiency  and  success  of  technical  training.  This  vector  intersects  with  the 
“Learner”  research  vector  from  the  standpoint  that  instructors  must  understand  and  draw  on 
trainees’  experiences  and  needs  as  learners  in  order  to  “optimize”  student  learning  experiences  in 
the  technical  training  environment. 

Research  is  needed  to  better  understand  how  instructors  and  instructional  methods 
contribute  to  success  or  failure  during  technical  training.  Such  knowledge  about  instructors  and 
instruction  could  inform  the  redesign  of  technical  training  so  as  to  improve  learner  engagement, 
comprehension,  and  successful  completion  of  training.  As  with  the  “Learner”  research  vector, 
research  into  instructors  and  instruction  has  the  potential  to  yield  both  short  and  long  term  impact 
by  identifying  areas  for  immediate  intervention  via  instructor  training  modifications  or  redesign, 
as  well  as  identifying  areas  where  incorporating  future  technologies  into  instruction  could 
improve  learning  outcomes. 

This  research  vector  should  also  explore  technical  training  through  the  lens  of  the 
instructor’s  preparation  and  practice.  This  dimension  of  inquiry  will  focus  on  understanding  the 
relationships  between  instructor  preparation/practice  and  student  performance  in  technical 
training.  One  subset  of  this  research  could  investigate  the  impact  of  instructor  selection  and 
preparation.  Another  subset  of  research  could  lie  in  examining  the  interactions  between 
instructors  and  individual  learners’  preferred  learning  modalities,  and  selection  of  instructional 
methods  as  they  impact  training  performance.  The  instructor  vector  of  research  has  the  potential 
to  uncover  leverage  points  of  intervention  to  reduce  training  time  while  improving  the  quality  of 
learning  and  graduation  rates. 

This  vector  is  interwoven  with  the  “Learner”  vector  in  that  research  in  psychology, 
biology,  linguistics,  and  neurophysiology  inform  our  understanding  of  the  human  brain  and 
learning,  and  provide  the  foundation  for  exploring  how  learning  in  technical  training 
environments  can  be  improved.  Instructor  preparation  and  professional  development  could 
incorporate  information  about  improving  human  learning  to  optimize  student  success.  The 
following  examples  are  provided  to  illustrate  research  opportunities  in  this  vector: 

■  What  knowledge  about  human  learning  needs  to  be  included  in  instructor  preparation  to 
optimize  student  motivation  and  learning? 

■  How  might  current  research  about  multi-tasking  inform  the  selection  and  application  of 
instructional  methodologies  in  the  technical  training  environment? 

■  How  might  the  relationships  between  instructors  and  learners  be  optimized  to  improve 
learning? 
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■  How  might  instructor  preparation  be  redesigned  to  better  meet  instructors’  learning  needs 
and  better  prepare  them  to  meet  individual  learner’s  needs  in  training? 

■  How  might  instructor  selection  and  preparation  be  redesigned  to  expedite  instructor 
readiness? 

■  What  preparation  must  instructors  receive  in  order  to  incorporate  digital  age 
literacy,  inventive  thinking,  effective  communication,  and  high  productivity  into  their 
instruction? 

From  these  examples,  it  is  clear  that  the  “Instructor”  vector  offers  a  rich  potential 
research  agenda,  particularly  as  it  intersects  with  the  “Learner”  vector  described  earlier. 

Research  suggests  that  instructors  are  the  most  influential  variable  in  learning.  Improving 
instructor  preparation  and  practice  is  one  of  the  most  promising  avenues  to  making  technical 
training  more  efficient  and  effective. 

3.3.3  Methodology 

This  research  vector  focuses  on  investigating  and  extending  current  methods  and 
practices  associated  with  the  design  and  development  of  Air  Force  technical  training  courses  for 
advanced  learning  environments  such  as  those  that  encompass  the  use  of  virtual  reality,  avatars, 
serious  games,  and  interactive  simulation  technologies  [Ref.  12].  The  overall  objective  of  the 
research  would  be  to  improve  our  knowledge  and  understanding  of  the  implications  (emphasis 
on  human  perspective)  of  advanced  learning  environments  in  order  to  better  inform 
methodologies  supporting  the  design,  development,  and  evaluation  of  Air  Force  technical 
training  courses.  The  primary  goals  of  this  research  would  focus  on  improving  the  overall  quality 
of  training  in  advanced  learning  environments  and  reducing  training  costs. 

Technology  is  being  rapidly  infused  into  development  of  training  systems  and 
applications  at  a  rapid  pace  -  a  pace  that  far  exceeds  the  introduction  of  new  educational 
paradigms  and  instructional  models/methodologies  that  provide  insight  on  how  to  best  harness 
the  advanced  capabilities  these  new  technologies  afford  for  the  purpose  of  education  and 
training.  For  the  most  part,  traditional  pedagogical  methods  are  still  in  use  to  design,  develop 
and  deliver  education  and  training.  The  question  is  how  well  suited  are  these  traditional  methods 
are  for  supporting  training  and  education  in  advanced  learning  environments?  In  what  manner 
do  methods  need  to  change  to  support  training  in  these  environments? 

The  scope  of  the  research  encompassed  by  this  vector  also  includes  the  analysis  of 
training  requirements,  development  of  training  objectives,  curriculum  design,  identification  and 
selection  of  instructional  methods,  and  instructional  material  development.  The  research  should 
be  focused  on  identifying  and  addressing  shortfalls  in  the  long-standing  (circa  1965) 

Instructional  Systems  Development  (ISD)  process  which  is  the  official  Air  Force  process  for 
determining  training  requirements  and  developing  education  and  training  courses  for  Air  Force 
personnel  (see  AFH  36-2235,  Volume  9).  Potential  research  questions  that  might  be  addressed 
as  part  of  this  vector  include  the  following: 

■  What  types  of  interdisciplinary  skills  are  needed  to  support  the  design,  development, 
and  evaluation  of  these  advanced  learning  environments? 

■  How  well  suited  are  current  instructional  methods  and  strategies  for  promoting 
learner  success  in  advanced  learning  environments? 
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■  What  type  methods  and  authoring  tools  are  needed  for  developing  and  updating 
instructional  materials  (content)  in  advanced  learning  environments? 

■  How  well  are  the  technologies  underlying  advanced  learning  environments  aligned 
with  particular  pedagogical  principles,  theories,  models,  hypothesis,  or  intuitions? 
[Ref.  12] 

■  What  is  the  impact  of  advanced  learning  environments  on  learning  and  retention  for  a 
growing  SLD  (specific  learning  disabilities)  population  with  dysgraphia,  dyscalculia, 
dyslexia,  dysnomia,  memory  &  processing  issues?  Same  question  for  a  more 
culturally  diverse  population  of  learners? 

■  What  do  we  know  about  the  social  /  cultural  needs  and  learning  styles  of  future  Air 
Force  trainees  (Millennials)  that  can  be  leveraged  to  better  inform  the  design  and 
development  of  technical  training  courses  (particularly  the  development  or  authoring 
of  content  and  selection  of  training  technologies)? 

■  How  relevant  are  existing  instructional  design  methods  or  processes  such  as  the  Air 
Force  ISD  process  for  designing  and  developing  course  curriculum  and  instructional 
materials  for  training  in  virtual  and  collaborative  learning  environments?  What 
shortfalls  exist  in  current  instructional  design  methods  or  processes  for  these 
environments? 

The  methodology  vector  intersects  with  other  research  vectors  discussed  in  this  report, 
particularly  the  learner,  evaluation,  instructor,  and  technology  vectors.  Processes  like  ISD  must 
somehow  take  into  account  the  knowledge,  skills,  abilities,  and  needs  of  individual  learners. 
These  factors  (or  attributes)  are  addressed  to  some  extent  in  the  research  outlined  for  the  learner 
vector,  particularly  from  the  standpoint  of  gaining  a  better  understanding  of  how  these  factors 
impact  training  performance.  Research  proposed  for  the  evaluation  vector  dealing  with  metrics 
and  methods  for  evaluating  advanced  training  environments  will  help  inform  the  development  of 
new  strategies  and  methods  for  designing  and  developing  instructional  materials  for  technical 
training  courses.  Research  discussed  as  part  of  the  technology  vector  dealing  with  understanding 
the  limitations  of  new  technologies  (hardware  and  software)  that  underpin  advanced  learning 
environments  from  both  a  human  and  technical  perspective  will  help  better  inform  the  process  of 
selecting  and  designing  instructional  materials  (content),  as  well  as  in  selecting  the  “best” 
method  for  delivering  technical  training  courses  to  trainees  (e.g.  in  advanced  distributed  learning 
environments).  This  will  be  extremely  important  to  consider  in  light  of  the  proposed  Air  Force 
MyBase  concept  discussed  earlier  in  this  report  that  seeks  to  leverage  existing  virtual 
environments  like  Second  Life  for  developing  future  training. 

3.3.4  Evaluation 

This  research  vector  focuses  on  exploring  and  applying  evaluation  resources  to  support 
the  development  of  criteria  for  evaluating  technology  applications  in  training.  Technology 
applications  must  enhance  trainees’  experiences  and  address  their  needs  as  learners  in  order  to 
optimized  learning  in  the  technical  training  environment. 

Research  is  needed  to  develop  and  test  evaluation  criteria  to  guide  decisions  about 
technology  applications  in  training  based  on  learners’  needs,  the  development  of  required 
knowledge  and  skills,  motivation,  and  engagement  as  contributors  to  success  or  failure  during 
technical  training.  These  criteria  could  then  inform  the  redesign  of  technical  training  so  as  to 
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improve  learner  engagement,  comprehension,  and  successful  completion  of  training.  This 
research  has  the  potential  to  yield  both  short  and  long  term  impact  by  identifying  immediate 
applications  of  evaluation  criteria  and  processes  to  inform  training  modifications  or  redesign,  as 
well  as  identifying  areas  where  applying  evaluation  criteria  to  future  technologies  may  improve 
learning  and  performance  outcomes.  In  addition,  this  vector  of  research  would  focus  on 
understanding  the  relationships  between  technology  applications,  instructor  expertise,  and 
learners’  physical,  psychological,  social,  and  cognitive  needs  in  the  context  of  training.  The 
evaluation  vector  of  research  has  the  potential  to  guide  decision-making  related  to  technology 
applications  so  as  to  reduce  training  time  while  improving  the  quality  of  learning  and  graduation 
rates. 

Given  the  importance  of  evaluation  to  guide  decision-making,  it  is  vital  to  have  a 
framework  of  standards  for  planning  and  judging  the  validity  of  any  evaluation.  Fortunately, 
such  a  framework  has  been  developed  by  the  Joint  Committee  on  Standards  for  Educational 
Evaluation.  This  framework  has  been  adopted  by  several  professional  associations,  including  the 
American  Association  of  School  Administrators  (AASA) ,  the  American  Counseling  Association 
(ACA),  the  American  Educational  Research  Association  (AERA),  the  American  Evaluation 
Association  (AEA),  the  American  Psychological  Association  (APA),  the  Canadian  Evaluation 
Society  (CES),  the  Canadian  Society  for  the  Study  of  Education  (CSSE),  the  National 
Association  of  Elementary  School  Principals  (NAESP),  the  National  Association  of  Secondary 
School  Principals  (NASSP),  the  National  Council  on  Measurement  in  Education  (NCME),  and 
the  National  Education  Association  (NEA).  This  framework  identifies  thirty  standards  divided 
into  four  subcategories:  utility  standards,  feasibility  standards,  propriety  standards,  and  accuracy 
standards.  These  standards  can  be  used  as  a  tool  for  evaluation  planning,  and  as  a  set  of  criteria 
by  which  any  evaluation  can  be  judged. 

Efforts  or  tasks  associated  with  identifying  existing  evaluation  resources,  interpreting 
them  for  use,  developing  new  evaluation  criteria  specifically  for  technical  training  contexts,  and 
testing  the  usefulness  of  evaluation  criteria  in  the  field  are  the  kinds  of  endeavors  that  should  be 
considered  as  part  the  Evaluation  research  vector.  Two  examples  are  provided  to  illustrate 
research  opportunities  as  part  of  this  vector: 

■  Guskey’s  five-levels  for  evaluating  professional  development  include:  1)  Participants’ 
reactions,  2)  Participants’  learning,  3)  Organization  support  and  change,  4) 
Participants’  use  of  new  knowledge  and  skills,  and  5)  Student  learning  outcomes. 

How  might  this  existing  model  of  evaluation  for  training  inform  the  development  of 
specific  criteria  to  evaluate  technology  applications  in  technical  training?  [Ref.  13] 

■  An  Educator's  Guide  to  Evaluating  the  Use  of  Technology  in  Schools  and  Classrooms 
-  December  1998  was  developed  for  the  U.S.  Department  of  Education  by  the 
American  Institutes  for  Research  in  conjunction  with  its  formative  evaluation  of  the 
Technology  Literacy  Challenge  Fund.  The  guide  represents  a  joint  effort  among  the 
Office  of  Educational  Research  and  Improvement,  the  Office  of  Educational 
Technology,  and  the  Office  of  Elementary  and  Secondary  Education.  An  approach  to 
developing  an  evaluation  is  provided  focusing  on  identifying  goals,  indicators, 
benchmarks,  measures,  and  gathering  evidence.  How  might  these  guidelines  inform 
the  development  of  an  evaluation  process  for  technology  applications  in  technical 
training? 
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This  vector  intersects  with  the  Learner,  Instructor,  Curriculum,  and  Technology  research 
vectors,  in  that  research  in  these  areas  inform  our  understanding  of  the  components  of  effective 
technical  training;  human  learning,  relationships  between  instructor  and  learner,  content,  and 
human  responses  to  various  levels  of  complexity  in  technology  applications.  These  aspects  of 
training  offer  insight  into  the  kinds  of  criteria  needed  to  determine  the  effectiveness  of 
technology  applications  in  technical  training  environments.  Even  though  the  dominate  goal  of 
technology  applications  in  training  is  to  increase  various  aspects  of  student  learning  and,  in  some 
cases,  instructor  performance;  many  variables  contribute  to  whether  the  use  of  a  particular 
technology  application  meets  this  goal.  Contributing  factors  related  to  these  other  research 
vectors  influence  evaluation  criteria.  Important  evaluation  questions  may  be  precursors  to 
determining  the  impact  on  student  and  instructor  performance  of  a  given  technology  application, 
for  example:  (a)  was  enough  technology  available,  and/or  was  access  sufficient  enough  to 
potentially  make  a  difference  in  performance?  (b)  were  instructors  adequately  trained  to  use  the 
technology  provided  and  (c)  were  technology  applications  appropriately  supported/integrated  by 
the  curriculum? 

The  ideas  and  questions  presented  above  are  just  a  few  examples  of  how  the  Evaluation 
vector  could  be  explored.  This  vector  offers  a  potentially  rich  research  agenda,  especially  at  the 
intersection  with  other  research  vectors  described  in  this  report.  Criteria  for  evaluating 
technology  applications  in  training  are  essential  to  ensure  that  technical  training  is  as  effective 
and  efficient  as  possible. 

3.3.5  Environment 

This  research  vector  focuses  on  better  understanding  the  role  of  the  learning  environment 
both  inside  and  outside  the  classroom,  including  factors  that  contribute  to  student  stress,  and 
formulating  recommendations  for  modifying  institutionally  related  processes,  practices,  and 
policies  that  could  improve  the  efficacy  and  success  of  technical  training. 

Research  is  needed  to  better  understand  the  dynamics  of  the  learning  environment  and 
how  the  interaction  between  students,  instructors,  delivery  modalities,  and  the  communication 
climate  affect  learning  and  knowledge  transfer.  Insight  into  the  in-residence  learning 
environment  could  give  schoolhouse  leadership,  career  field  managers,  and  curriculum  designers 
a  better  perspective  on  this  generation’s  students.  This  research  has  the  potential  to  yield  both 
short  and  long  term  impact  by  identifying  environment  related  shortfalls  that  could  be  addressed 
through  immediate  or  near  term  interventions.  Examples  include  examining  the  length  and 
structure  of  the  duty  day  for  technical  training  including  the  sequencing  of  course  instruction  and 
other  activities  that  reduce  instructor/student  stress  and  workload. 

This  vector  is  interwoven  with  all  other  training  dimensions.  The  learning  environment  is 
fluid  and  self-generating,  based  upon  curriculum  delivery  method,  selected  technologies,  learner 
behavior,  motivation,  and  knowledge,  and  instructor  experience  and  attitude.  This  presents  a 
challenge  to  researchers  attempting  to  study  this  aspect  of  learning  because  a  “learning 
environment”  is  comprised  of  “all  of  the  physical  surroundings,  psychosocial  or  emotional 
conditions,  and  social  or  cultural  influences’’  present  in  a  learning  situation  [Ref.  14].  Therefore, 
research  in  this  area  could  quite  naturally  focus  on  the  intersections  between  the  training 
dimensions  that  shape  the  learning  environment.  Such  a  holistic  approach  would  serve  to 
highlight  those  points  at  which  Millennial  students,  military  indoctrination,  instructor  quality, 
prerequisite  knowledge,  societal  norms,  game-influenced  neurophysiology,  and  institutional 
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goals  are  at  tension.  The  following  examples  are  provided  to  illustrate  research  opportunities  in 
this  vector. 


■  What  types  of  communications  climate  exist  in  a  “typical”  tech  training  class?  What  is 
the  level  of  instructor- student  interactivity?  What  are  the  norms  for  verbal,  non-verbal, 
and  para- verbal  communication?  Is  there  a  generational  trend? 

■  How  does  the  institutional  goal  of  “military  indoctrination”  affect  the  learning 
environment?  What  coaching  practices  might  be  implemented  to  create  an  environment 
that  is  disciplined  without  being  oppressive? 

■  Is  the  institutional  investment  in  the  training  environment  -  e.g.  modern  technology,  hard 
trainers  are  well  maintained  and  fit  for  purpose,  facility  upkeep  -  in  consonance  with  the 
“valuing”  of  the  students  as  Air  Force  members  doing  an  important  job. 

■  How  does  the  extended  duty  day  affect  student  learning  and  cognition?  What  schedule  or 
task  changes  could  reduce  student  stress  and  enhance  attention,  participation,  and 
performance? 

■  How  do  instructors  treat  poor  performing,  female,  or  minority  students?  Are  these 
students  encouraged  to  ask  questions;  do  they  receive  positive  support?  Do  instructors 
call  students  by  name?  What  techniques  do  instructors  use  to  elicit  student  feedback? 

■  Does  the  curriculum  and  instructor  behavior  encourage  investigation  and  risk  taking? 

Are  students  allowed  to  “fail-to-success”  as  part  of  the  instructional  process? 

■  Is  the  physical  classroom  environment  conducive  to  learning?  Does  each  student  have 
enough  space?  Are  there  physical,  sensorial,  or  other  environmental  factors  that  inhibit 
learning? 

■  Does  the  students’  living  environment  and  off-duty  time  management  choices  affect  their 
retention  and  performance? 

■  Are  high-performing  or  pre-qualified  (e.g.  technical  college  or  an  experienced  tradesman) 
students  encouraged  to  actively  share  their  knowledge  in  the  class?  Do  instructors 
properly  leverage  these  students  to  motivate  and  assist  poor  performers? 

■  Are  expectations  for  intra-student  support,  study  groups,  and  other  mutually  beneficial 
learning  behaviors  clearly  differentiated  from  cheating?  Are  individual  or  group 
performance  tests  more  meaningful  in  a  technical  training  environment  than  “closed  book 
multiple  choice”  tests?  [The  relevancy  here  is  that  AETC  schools  hector  students  from 
the  first  day  about  cheating.  In  the  “real  Air  Force”  there  is  no  such  thing  as  a  “closed 
book”  task  -  tech  data  is  mandatory  and  failure  to  use  it  is  a  punishable  offense.] 


From  these  few  examples,  it  is  clear  that  the  Environment  vector  offers  a  rich  potential 
research  agenda,  and  it  intersects  with  the  other  potential  vectors  described  in  this  report.  The 
student  environment  should  be  supportive  and  effective  for  learners  of  varied  capabilities,  social 
skills,  and  prerequisite  knowledge.  Research  into  this  dynamic  factor  in  the  learning  experience 
may  illuminate  simple  changes  to  practices  and  policies  that,  while  rooted  in  culture  and 
tradition,  fail  to  produce  a  mission-capable  warfighter. 
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3.3.6  Technology 


This  vector  is  focused  on  developing  a  better  understanding  of  the  relationships  between 
the  use  of  technological  “tools”  (hardware  and  software),  students,  and  their  instructors. 
Technology  itself  is  a  broad  area  and  encompasses  several  attributes  that  are  pertinent  to  training 
(e.g.  hardware,  software,  networking,  etc.),  so  while  the  discussion  in  this  section  is  fairly 
extensive,  it  is  limited  to  topics  and  questions  that  were  considered  to  be  within  the  scope  of 
potential  research  interest  to  AETC  and  AFRL. 

The  approach  here  is  first  to  summarize  various  technology  attributes  and  comment  on 
the  impact  those  attributes  may  have  on  learners  and  instructors.  These  technology  attributes 
also  precipitate  various  evaluation  issues.  Finally,  there  are  some  research  questions  which  arise 
that  are  related  to  the  individual  learner,  instructor,  or  evaluation  issues,  or  are  suggested  by  the 
interrelationships  between  these. 

■  The  technology  types  considered  are  hard  (physical)  versus  soft  (computer-based) 
systems. 

o  For  learners,  hard  trainers  have  the  advantage  of  physicality  but  a  limitation  of 
specificity.  Soft  trainers  promote  a  variety  of  experiences,  allow  interactive 
guidance,  and  exercise  abstract  reasoning  processes. 

o  For  instructors,  hard  trainers  promote  fixed  lesson  plans,  hands-on  training, 
but  have  the  limitations  of  being  dated  and  difficult  or  costly  or  even 
impossible  to  maintain  (such  as  a  complete  aircraft).  Software  trainers  are 
often  rigid  (limited  options  or  lessons)  or  brittle  (unexpected  failures  or  bugs), 
and  difficult  to  modify  or  update  once  deployed. 

o  Evaluations  differ  by  type:  trainees  work  directly  on  hard  trainers,  but  cause 
wear  and  tear  and  incur  possible  injuries  (as  in  real  life);  while  soft  trainers 
permit  ready  data  collection  but  sacrifice  the  physical  experience  and  manual 
task  control.  Software  promotes  computer-based  testing  for  concept 
understanding  and  skill  transference. 

o  Possible  research  question:  Software  trainers  can  be  of  value  when  the 
problem  demands  some  level  of  abstract  reasoning  and  problem  flexibility, 
but  demand  more  time  and  effort  cost  for  updates.  What  process  can  be  used 
to  evaluate  whether  a  software  training  investment  is  effective  and  efficient 
from  cost  and  learning  perspectives? 

■  The  technology  hardware  scene  is  rapidly  evolving  with  new  options  in  mobility  and 
processing  power. 

o  For  learners,  technology  should  be  easy  to  use,  light  weight,  possibly  portable 
and  wireless,  and  multi-functional  to  avoid  gadget  proliferation.  Devices  must 
adequately  support  screen  resolution  and  data  storage  requirements  for 
anticipated  schematics  and  diagrams,  and  must  be  visible  and  readable  in  the 
learning  environment. 

o  Instructors  likely  prefer  inexpensive  technology  to  mitigate  device  loss  or 
damage,  require  flexible  content  use  and  update,  and  may  want  multi-modal 
capability  for  voice  or  hands-free  operation.  In  addition,  instructors  may 
leverage  opportunities  for  two-way  communication  for  questions,  assignment 


19 


submission,  and  collaborations.  Such  devices  may  also  hold  potential  for 
multiple  natural  language  support  and/or  translation, 
o  Evaluating  technology-based  learning  devices  includes  research  questions 
such  as  determining  appropriate  information  presentation  density  “per  square 
inch”  and  studying  the  actual  learning  transfer  relative  to  other  media  (paper, 
hard  trainer,  real  system). 

■  System  architectures  must  tale  into  account  existing  hardware  and  communication 
structures. 

o  From  a  learner’s  perspective,  “small”  device  capabilities  are  increasing; 
leading  to  “smarter”  portable  devices  with  wireless  download,  3D  graphics, 
video,  and  multi-modal  interfaces.  Local  software  and  interactive  systems 
minimize  the  need  for  constant  connectivity  (e.g.  local  server  or  the  Internet), 
but  connectivity  is  already  ubiquitous,  and  nearly  free  for  portable  devices. 
Thus  client-server  systems  offer  good  trade-offs  for  updating  content  and 
enabling  collaboration. 

o  Likewise,  the  instructor  can  access,  control,  and  update  server-side  content, 
which  must  be  pushed  to  learners’  devices  though  extant  wireless  connectivity 
or  even  memory-stick  transfer.  The  instructor  can  access  and  receive 
uploaded  information  from  student  devices  for  evaluation, 
o  For  learner  evaluation,  the  line  between  Internet  and  local  networks  are 
blurred  at  the  device  level,  meaning  that  content  and  individual  assessments 
can  be  conducted  through  a  server-side  interface, 
o  The  research  questions  here  involve  (a)  the  need  to  determine  the  most  likely 
wireless  computing  environment  compatible  with  selected  devices,  course 
content  and  learning  goals,  and  (b)  how  to  best  design  and  develop  the 
software/communications/hardware  architecture  to  support  mobile  learning. 

■  Given  that  the  requirements  for  many  learning  tasks  involve  a  visual  component, 
technology  display  properties  are  highly  relevant  to  the  teaching  process. 

o  For  the  learner,  classroom  active  whiteboards  have  not  been  very  successful. 
Learners  possess  mobile  phones,  but  no  common  platform  is  mandated. 
Laptops  are  expensive  and  less  portable  for  field  use.  Augmented  reality 
displays  (wearable  eyeglass  mounted  sliver  displays)  for  content  overlay  are 
lightweight,  attractive  and  relatively  inexpensive  for  information  presentation 
in  context,  but  require  a  wearable  computer  or  future  portable  device 
communications  link  (such  as  Bluetooth).  Voice  control,  if  reliable,  would  be 
useful  for  hands-off  procedure  stepping.  Also,  voice  input/output  may  be 
difficult  in  noisy  environments,  so  one  may  need  to  utilize  some  wired  or 
wireless  COM  link. 

o  Display  technology  is  important  to  instructors,  as  it  is  the  primary 

communication  channel  for  visual  instructional  materials.  While  classrooms 
have  projection  equipment,  they  are  not  necessarily  available  in  hangers  or  on 
the  flightline.  Portable  display  devices  allow  pushing  content  to  the  point  of 
use,  while  augmented  reality  display  systems  can  even  move  information  into 
proximity  (registration)  with  physical  systems, 
o  Evaluation  techniques  might  test  if  task  steps  translate  into  proper,  faster  or 
better  part  location,  positioning,  removal,  tool  use,  etc.  by  comparing  results 
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with  and  without  “on-site”  visual  aids  (e.g.  augmented  reality  to  minimize 
errors). 

o  Accordingly,  some  research  questions  include  determining  (a)  what  learning 
dimensions  are  best  served  by  which  display  technology,  (b)  studying 
inexpensive  augmented  reality  for  training/jobsite  Technical  Order 
presentation  and  task  views,  location,  and  process  visualization,  and  (c)  what 
displays  are  best  for  hazard  mitigation  and  training  with  respect  to  cautions 
and  warnings? 

■  Software  design  and  implementation  are  the  major  human  capital  expenses  and 
bottlenecks  in  developing  instructional  systems. 

o  From  the  learner’s  perspective,  canned  demos  are  rigid,  specific,  and  lack 
generalizability  (limiting  concept  formation).  Interactive  Computer-Based 
Training  (CBT)  is  an  industry  standard,  being  primarily  workstation-based, 
self-paced,  oriented  toward  specific  outcomes,  but  usually  lacking  connection 
with  physical  systems  that  are  the  subject  of  maintenance  instruction.  Shared 
virtual  environments  such  as  Second  Life  and  the  proposed  Air  Force  MyBase 
encourage  exploration,  but  may  become  unstructured  if  not  carefully 
designed.  These  systems  encourage  communication  for  information  and 
problem-solving,  and  thus  also  open  possibilities  (and  problems)  for  social 
networking  and  peer  collaboration.  Contemporary  learners  (Millennial)  also 
seem  to  be  motivated  by  games:  interactive  systems  that  capitalize  on  human 
characteristics  to  explore,  experience,  seek  goals,  obtain  rewards,  and 
challenge  competitors  through  rapid  response,  strategic  adaptation  and  a 
desire  to  “win.”  Some  “games”  may  utilize  simulations  in  a  standalone 
system  or  as  an  adjunct  to  a  game  or  collaborative  environment.  These 
simulations  provide  more  open-ended  exploratory  and  learning  experiences 
with  varying  possibly  outcomes  (with  the  possibility  to  learn  from  failures), 
o  For  instructors,  canned  demonstrations  possess  limited  explanatory  power. 
Interactive  CBT,  though  an  industry  standard,  appears  not  well  integrated  into 
the  existing  Air  Force  schoolhouse  mindset  and,  moreover,  is  less  suitable  for 
physical  training  tasks.  A  virtual  environment  such  as  MyBase  may  encourage 
exploration,  but  must  be  carefully  designed  and  aware  that  there  are  unknown 
consequences  of  social  communication  in  these  environments.  Authoring 
games  will  likely  require  time  and  expertise  beyond  normal  instructor  skill 
and  energy  commitments  and  thus  involve  external  (contractor)  development 
support.  Likewise,  simulations  whether  part  of  a  game  or  standalone,  would 
most  likely  also  require  external  development  support, 
o  Evaluating  learning  in  computer-based  systems  is  highly  quantitative  and 
includes  a  range  of  evaluation  metrics  from  binary  correctness  to  game 
timing/score  to  depth  to  experiences  attempted, 
o  Research  questions  include  (a)  what  are  the  features  that  help  determine  a 
“best”  computer  implementation  platfonn  for  a  desired  learning  outcome,  (b) 
what  are  existing  data  points  that  can  help  us  understand  the  shape  and  slopes 
of  student  learning  performance  in  this  space,  and  (c)  what  would  be  the  best 
software  structure  learning  environment  in  which  potential  washbacks  and 
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failures  can  be  detected  and  mitigated  though  additional  experiences, 
interventions,  or  social  context? 

■  Network  technology  allows  multiple  users  to  share  the  same  space,  data,  or 
experiences. 

o  For  the  learner,  networking  opens  doors  to  collaboration  and  teamwork  rather 
than  isolated  individual  action.  Individual  instruction,  skill  assessment  and 
self-awareness  are  still  important,  but  collaboration  is  important  to  the  Air 
Force  mission  and  involves  teamwork,  group  problem-solving,  and  peer 
mentorship.  The  option  of  using  social  networking  capabilities  supported  by 
Second  Life  or  the  proposed  MyBase  allows  exploration  of  a  large  possibly 
unstructured  space,  but  also  opens  the  door  to  undesirable  outcomes  such  as 
time  wasted,  time  lost,  missed  opportunities,  failure  to  achieve  goals,  etc.  One 
option  is  to  use  virtual  teammates,  both  to  encourage  collaboration  and  task 
focus,  but  also  to  allow  team  activities  without  the  entire  team  being 
physically  present.  This  opens  up  24/7  task  rehearsal  and  practice,  removes 
the  stigma  of  failure  in  a  group,  and  increases  understanding  of  the  need  for 
and  communication  with  other  members  of  the  team, 
o  For  the  instructor,  one  student  at  a  time  means  maximum  attention  for  that 
student,  but  possibly  less  efficiency  in  the  classroom  as  a  whole.  A 
collaboration  experience  could  be  valuable  if  that  experience  produces  the 
desired  learning  outcomes  or  satisfies  specific  training  objectives,  as  it 
exposes  language,  communication  and  coordination  issues  up  front  during  the 
learning  phase.  In  this  regard,  the  virtual  Second  Life  approaches  are 
currently  unproven  for  learning  physical  skills  or  collaborative  physical  tasks. 
It  is  known,  however,  that  virtual  adversaries  and  teammates  are  of  proven 
value  in  military  operational  simulations, 
o  Evaluation  would  depend  primarily  in  learning  transfer  to  the  physical 
individual  or  teamwork  environment  and  situation.  One  could  measure 
quality  and  content  in  communications  before  and  after  training  experience, 
o  Research  questions  must  address  the  value  of  virtual  team-members  to 

training  for  multi-person  skills  (maintenance)  as  opposed  to  the  planning  and 
real-time  execution  of  operations  such  as  those  in  America ’s  Army.  Some 
positive  outcomes  are  appearing  from  the  interactions  of  learners  with  virtual 
collaborators  or  adversaries  in  projects  at  the  Institute  for  Creative 
Technologies  (ICT)  at  USC. 

■  Technology  offers  options  for  the  user ’s  role  or  engagement  in  the  learning  process. 

o  The  learner’s  role  in  passive  (3ld  person)  activities  include  watching  some 
activity  with  the  possibility  of  changing  or  interrupting  it.  Early  CBT  systems 
used  canned  or  branching  video  to  allow  decision  points  within  a  fixed 
prepared  script.  Many  computer  games  fall  into  this  category  as  well.  By 
placing  the  learner  in  an  active  (1st  person  or  avatar)  experience,  feelings  of 
presence  and  control  in  the  environment  may  be  accentuated.  These 
experiences  allow  the  subjects  a  real  life  visual  perspective,  and  enhances 
understanding  of  their  physical  relationship  with  the  surroundings.  When 
combined  with  simulations,  the  experience  can  be  relatively  flexible  and 
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open-ended.  The  presence  of  some  structured  story  line,  guide,  or  coach  is 
useful  for  managing  the  interaction  and  keeping  the  student  on  task. 

o  From  the  instructor’s  perspective,  the  advantage  of  the  active,  first  person 
approach  is  the  direct  connection  to  the  experience:  doing  rather  than  just 
watching.  The  advantages  to  the  passive  3rd  person  view  are  being  able  to 
perceive  the  whole  body’s  relationship  to  space,  structures  and  other  people, 
and  to  visualize  errors  as  those  of  someone  else.  Typical  assessments  should 
compare  these  two  types  of  experiences. 

o  Research  questions  include:  (a)  how  can  we  develop  test  suites  to  assess  and 
compare  these  two  types  of  experiences  and  (b)  can  computer  games  be  used 
to  assess  an  individual’s  possible  vocational  skills  by  observing  selection  and 
performance  across  a  representative  sample  of  genres? 

Technology  allows  a  variety  of  channels  for  user  inputs  to  a  learning  system. 

o  One  can  expect  the  Millennial  learner  to  be  relatively  facile  with  standard 
mouse,  cursor,  joystick,  and  keyboard  input  devices.  Voice  is  useful  for 
hands-free  operation,  but  difficult  to  justify  in  noisy  operating  environments, 
with  speech  or  language  variants,  or  with  potential  high  understanding  error 
rates.  Motion  capture  has  potential  for  teaching  highly  specific  physical 
behaviors  (such  as  surgery,  possible  Improvised  Explosive  Device  (IED) 
disposal,  etc.),  but  is  relatively  expensive  to  procure  and  awkward  to  manage. 
Very  recent  input  device  technology  includes  the  Wii  physical  interface  which 
is  a  cross  between  a  game  controller  and  two-hand  motion  capture.  The  Wii 
controller  has  some  potential  to  teach  primitive  manipulative  aspects  of 
physical  skill.  Video  (non-suited)  motion  capture  is  in  the  future  (beyond 
2010)  but  has  the  potential  to  collect  physical  performance  data  “non- 
invasively.”  Synthetic  control  panels  or  hard  simulators,  such  as  vehicle  or 
airplane  cockpit  simulators,  have  seen  limited  application  to  maintenance  or 
material  handling  tasks. 

o  From  the  instructor’s  perspective,  standard  mouse,  cursor,  joystick  and 
keyboard  manipulation  may  be  assumed  in  the  learner’s  skill  set,  but  text 
inputs  are  subject  to  varying  language  skills.  Voice  control  appears  awkward 
except  for  situations  where  the  learner  is  already  using  a  COM  link  and  must 
communicate  with  team  members  or  instructor.  Motion  capture  would  be 
outside  the  scope  of  typical  instructor  expertise  and  would  slow  down  and 
even  get  in  the  way  of  the  physical  experience.  Motion  capture  could  be  used, 
however,  to  develop  “gold  standard”  actions  by  capable  instructors  to 
illustrate  physical  tasks  since  they  allow  3D  motion  animations  that  can  be 
examined  from  any  vantage  point,  rather  than  the  single  point  of  view  of  a 
video.  Video  motion  capture  (for  3D  data)  is  too  technologically  poor  at  this 
time  to  be  considered  an  alternative  to  physical  motion  capture;  moreover 
video  capture  requires  more  than  one  camera  and  a  clear  line  of  sight  to  the 
subject  for  each  camera.  A  synthetic  trainer  for  physical  tasks  would  be 
equivalent  to  creating  the  “Holodeck”:  the  real  limitation  is  the  inability  of 
virtual  scenes  to  impose  large  scale  physical  forces  and  torques  on  a  subject  - 
and  even  if  they  could,  one  must  be  wary  of  the  concomitant  danger  of  doing 


o  Learner-side  evaluations  seem  to  require  focus  on  skill  transfer  from  familiar 
computer  input  tools  to  the  necessity  of  physical  experience.  Tool  skill 
acquisition  in  a  virtual  space  must  be  tested  on  hard  devices.  Comparing  a 
learner’s  physical  performance  to  an  instructor’s  “gold  standard”  via  motion 
capture  and  comparative  measurement  can  be  reserved  for  certain  critical 
tasks,  but  the  software  to  do  these  comparisons  must  still  be  developed, 
o  Research  questions  include  (a)  can  motion  capture  and  motion  comparison  be 
profitably  used  to  train  or  even  familiarize  students  with  certain  precision 
physical  skills  that  may  be  required,  (b)  can  the  Wii  controller  be  used  for 
basic  tool  selection  and  appropriate  use,  and  (c)  can  learner  performance  in 
games  and  simulations  be  used  to  help  better  predict  career  directions  and  skill 
affinities? 

■  Given  the  potential  for  a  technological  learning  environment,  an  instructor ’s  role , 
effort,  and  engagement  may  change. 

o  Passive  material  preparation  in  advance  for  some  form  of  visual  or  CBT 
delivery  is  the  present  norm.  Active  learning  methods  may  require  that  the 
instructor  be  accessible  via  online  system  or  through  an  avatar  for  outside 
class  contact.  The  cost  of  this  accessibility  is  usually  prohibitive  (as  well  as 
personally  invasive),  so  there  has  been  an  emergence  of  systems  for  Guided 
Experiential  Learning  developed  at  ICT  and  under  development  at  the 
University  of  Pennsylvania,  for  example.  These  systems  offer  virtual  coaches, 
mentors,  or  aides  that  can  guide  or  suggest  behaviors,  allow  a  safe 
environment  in  which  to  make  errors,  or  step  in  to  help  redirect  errors.  The 
development  costs  of  these  systems  are  justified  if  military,  health  and  social 
goals  are  important  enough. 

o  The  evaluation  of  such  instructor  surrogate  systems  is  not  fully  mature 
However,  one  can  measure  time  for  task,  error  rates,  and  analyze  questions 
posed  to  the  virtual  instructor  and  subjectively  judge  the  quality  of  the 
answers  or  guidance  received  (if  any), 
o  There  are  still  open  research  questions  on  the  use  of  virtual  agents  or 

instructors  for  training  and  learning.  They  can  be  invasive  and  annoying  (e.g. 
the  infamous  Microsoft  paperclip  character).  So  one  question  to  be  addressed 
is  how  is  a  virtual  instructor  best  utilized?  What  learning  outcomes  are 
affected  (positively  or  negatively)  by  the  existence  of  a  virtual  instructor? 

Can  virtual  instructors  be  extended  to  on-the-job  training  applications  as  well 
as  schoolhouse  use? 

■  Technology  alone  does  not  provide  a  learning  environment;  it  must  be  supported  by 
appropriate,  in-depth,  and  carefully  authored  content. 

o  Individual  instructor  authoring  choices  may  be  very  limited,  such  as  paper, 
text,  PowerPoint,  or  Flash  materials.  Communities  of  instructors  may  develop 
materials  on  websites  or  wikis.  In-house  or  external  contractors  may  be 
needed  for  content  authoring  for  some  CBT  systems  involving  computer 
programming.  Contractor  support  will  very  likely  be  required  for  specialized 
games  and  simulation  tools,  as  well  as  virtual  environments  such  as  MyBase. 
o  For  course  evaluation,  and  authored  content  must  correlate  with  curricular 
goals  and  instructor  capabilities;  any  mismatch  will  hurt  “programmatics”  or 
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content  delivery  and  hence  learning  success.  Rather  than  a  least  common 
denominator  approach,  appropriate  authoring  tools  must  be  utilized  to  achieve 
learning  goals  given  the  available  or  desired  delivery  technology, 
o  Research  questions  here  include  (a)  what  Air  Force  institutions  and  instructor 
training  materials  need  to  be  modified  to  achieve  more  agile  content 
generation  goals,  (b)  what  role,  if  any,  do  instructors  have  as  it  pertains  to 
content  authoring  in  advanced  learning  environments,  (c)  if  they  do  have  a 
role,  then  how  can  future  technological  systems  enhance  content  authoring  at 
the  instructor  level,  and  (d)  how  can  we  measure  instructor  effectiveness  in 
the  transformation  of  learning  and  curricular  goals  into  technology-based 
instructional  materials? 

■  Once  content  is  authored,  it  may  not  be  permanent  or  static,  content  changes  and 
evolves  over  time  as  systems  are  modified  or  upgraded,  procedures  are  refined,  etc. 
o  This  presents  issues  for  instructors,  as  they  may  have  no  access  or  ability  to 
change  the  stored  content.  Systems  that  use  databases  or  wikis  may  be  more 
flexible.  Software  systems  built  by  external  contractors  may  be  fixed  or 
brittle  to  change  and  may  present  expensive  upgrade  or  correction  costs, 
o  The  potential  need  for  content  updates  should  be  part  of  the  evaluation  process 
for  technologically-based  learning  systems.  Review  processes  are  needed  to 
ensure  content  consistency  and  accuracy,  to  organize  and  maintain  responses 
to  frequently  asked  questions,  to  develop  novel  group  exercises,  or  to  exploit 
existing  and  emerging  technological  features, 
o  A  desirable  research  goal,  therefore,  is  to  ensure  that  technology-based 
authoring  tools  are  accessible  and  content-updateable  by  instructors.  Are 
there  authoring  systems  being  overlooked  or  other  options  on  the 
technological  horizon?  Can  quantifiable  benefits  from  guided  experiential 
learning  be  transitioned  to  lower  cost,  maintainable,  extensible  systems? 

Whatever  technological  approaches  to  learning  are  investigated,  we  must  always  be 
cognizant  of  the  safety,  policy,  social  and  regulatory  issues  associated  with  any  technology 
adoption  or  use.  It  is  also  important  that  decisions  regarding  the  adaptation  or  development  of 
technologies  for  applications  in  training  consider  the  return  on  investment  not  only  from  a  cost 
perspective  (including  reduced  training  time),  but  also  with  consideration  for  helping  better 
achieve  or  enhance  desired  learning  outcomes. 

3.4  Future  Training/Learner  Pilot  Study 

In  addition  to  the  workshops  and  other  research  activities  discussed  above,  a  small  pilot 
study  was  undertaken  at  the  82nd  TRW,  Sheppard  AFB,  TX  to  better  understand  the  learning 
preferences  of  Air  Force  Technical  Training  trainees  so  as  to  recommend  modifications  and 
redesign  of  technical  training  to  improve  engagement,  comprehension,  and  successful  course 
completion.  The  details  and  results  of  this  study  are  described  more  fully  in  the  report  titled 
“21s’  Century  Training  for  21st  Century  Learners”  included  as  Appendix  D  to  this  final  report. 

A  summary  of  the  pilot  study  goals,  study  approach,  and  results  are  summarized  in  this  section. 

The  goals  of  the  pilot  study  were  informed,  in  part,  by  the  data  collection  and  workshop 
activities  conducted  as  part  of  the  DO-9  research  effort.  These  goals  included  the  following:  a) 
determining  whether  current  Air  Force  Technical  Trainees’  are  21st  century  learners  -  based  on 
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attributes  and  characteristics  discovered  through  our  literature  search;  b)  assessing  whether  Air 
Force  technical  training  is  aligned  with  trainees’  learning  preferences  and  identifying  areas  of 
potential  improvement;  and  c)  identifying  leverage  points  in  training  where  innovations  in 
technology  could  potentially  improve  training  outcomes.  To  accomplish  these  goals,  a  review  of 
literature  was  conducted,  a  Technical  Training  Observation  Rubric  (TTOR)  was  developed  for 
use  in  technical  training  classroom  observations  of  teaching  methods,  and  responses  to 
“preference”  inventories  (based  on  the  Grasha-Reichman  learning  and  teaching  style  inventories) 
were  collected  from  trainees  and  instructors  respectively. 

The  Learning  Styles  Inventory  (LSI)  was  administered  to  172  trainees  at  Sheppard  AFB 
representing  six  different  technical  training  courses.  Responses  on  the  LSI  supported  the 
generalization  that  AF  technical  trainees  are  21st  century  learners  with  a  mean  LSI  composite  of 
Participant/Dependent/Collaborative.  The  Teaching  Styles  Inventory  (TSI)  was  administered  to 
22  technical  training  instructors  in  six  courses.  Instructor  responses  indicated  they  preferred 
Teaching  Style  Cluster  2-  Personal  Model/Expert/Formal  Authority  (see  attached  report)  with 
secondary  preferences  for  the  Facilitator/Delegator  styles.  Grasha’s  research  suggests  that 
Teaching  Styles  Cluster  2  best  supports  students  with  the  Participant/Dependent/Collaborative 
learning  styles  composite. 

The  FSI-TSI  data  indicated  there  was  alignment  between  instructors’  preferred  teaching 
cluster  and  trainees’  learning  preferences.  However,  responses  on  the  technology  use  items 
(added  to  the  ESI  and  TSI  for  this  pilot  study)  indicated  that  trainees  preferred  more  technology 
use  in  instruction  than  did  their  instructors.  In  addition,  while  the  ESI  and  TSI  data  indicated 
alignment  between  instructors’  and  trainees’  preferences,  classroom  observation  ratings  using  the 
TTOR  indicated  instructors  were  not  observed  teaching  in  their  preferred  Teaching  Styles 
Cluster.  Observed  instruction  was  not  representative  of  21st  century  methods. 

The  findings  of  this  pilot  study  suggested  that  technical  training  instruction  could  be 
better  aligned  to  meet  both  instructors’  preferred  teaching  styles  and  trainees’  learning 
preferences,  and  these  findings  are  discussed  in  more  detail  in  attached  pilot  study  report.  One 
point  worth  citing  here  is  that  the  study  did  indicate  that  there  is  some  alignment  between 
instructors’  and  trainees’  preferences  -  which  is  good  news,  in  that  it  suggests  that  instructors 
possess  the  desire  and  inclination  to  teach  in  ways  that  support  21st  century  learners’  preferences 
for  participation,  self-direction,  competence  and  collaboration.  However,  observed  instruction 
did  not  utilize  recommended  21st  century  practices  such  as:  cooperative  learning,  role  modeling, 
illustrations,  problem-solving,  multiple  examples,  discussions  of  alternative  approaches,  sharing 
of  thought  processes  for  obtaining  answers,  sharing  of  personal  experiences,  coaching  and 
guiding  with  feedback,  or  visual  tools. 

4.0  CONCLUSIONS  AND  RECOMMENDATIONS 

This  research  effort  provided  an  excellent  opportunity  for  developing  a  better 
understanding  of  the  current  and  future  challenges  confronting  the  Air  Force  technical  training 
environment.  While  the  scope  of  the  research  and  data  collection  focused  primarily  on  technical 
training  activities  at  the  82nd  TRW  at  Sheppard  AFB,  TX,  the  technical  training  challenges  cited 
in  this  report  are  arguably  common  to  the  challenges  faced  by  other  Air  Force  locations 
conducting  technical  training  (e.g.  Keesler  AFB,  Fackland  AFB,  etc.).  Therefore,  the  research 
questions  and  topics  discussed  in  this  report  are  intended  to  be  applicable  to  (or  have  some 
implications  for)  the  Air  Force  technical  training  community  as  a  whole.  As  stated  earlier,  it  is 
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important  to  recognize  some  of  the  training  challenges  and  needs  addressed  in  this  report  are 
significantly  impacted  by  factors  that  will  most  likely  not  be  resolved  or  addressed  through 
research  alone.  Budgetary  constraints  (e.g.  replacing  or  upgrading  training  equipment,  trainers), 
current  operational  tempo  (e.g.  pressure  to  increase  throughput  of  students  to  support  on-going 
operations),  and  current  organizational  and  training  policies  and  procedures,  are  factors  that  also 
need  to  be  considered  as  part  of  formulating  a  future  research  agenda  for  technical  training. 

The  report  proposes  six  vectors  of  research  for  technical  training  and  potential  research 
questions  and  topics  for  each  vector  that  are  deemed  important  to  the  development  of  a  research 
agenda  for  technical  training.  These  vectors  include:  Learners,  Methodology,  Instructor, 
Environment,  Evaluation,  and  Technology.  While  the  research  questions  and  topics  raised  for 
each  vector  are  considered  important  in  the  development  of  a  research  agenda  for  technical 
training,  the  vectors  cited  in  this  report  are  not  totally  independent.  There  is  interdependence 
between  some  vectors  in  terms  of  the  questions  and  topics  raised  that  should  be  considered  in  the 
development  of  a  research  agenda.  For  instance,  the  evaluation  of  training  technologies 
supporting  the  development  of  training  for  deployment  in  advanced  learning  environments  such 
as  proposed  by  the  MyBase  vision  will  require  suitable  metrics  for  evaluating  there  impact  on 
learning  outcomes,  cost  effectiveness,  and  current  instructional  systems  design  and  development 
processes.  These  type  metrics  and  criteria  currently  do  not  exist  for  effectively  evaluating 
learning  outcomes  (and  cost  effectiveness)  in  advanced  learning  environments  such  as  those 
utilizing  virtual  reality  and  incorporating  the  use  of  avatars.  Hence,  to  effectively  evaluate 
technologies  for  specific  technical  training  applications  in  these  environments,  a  research  agenda 
might  first  start  with  identifying  relevant  metrics  and  methods  for  collecting  data  and  measuring 
learning  outcomes  against  these  metrics.  The  next  step  might  be  to  conduct  research  using  these 
metrics  in  specific  training  contexts  utilizing  advanced  training  technologies  to  assess  the  impact 
on  learning  outcomes,  degree  of  accommodation  for  different  learning  styles,  as  well  as  to 
inform  the  refinement  of  metrics,  and  impact  on  current  curriculum  development  processes  (e.g. 
ISD  process),  instructor  training,  etc. 

Another  point  worth  noting  is  that  not  all  the  questions  and  topics  involve  long-term 
research  efforts.  For  example,  the  pilot  study  accomplished  as  part  of  this  report  indicated  that 
while  there  appears  to  be  alignment  between  the  learning  styles  of  trainees  (21st  century 
learners),  and  the  preferred  teaching  styles  of  instructors  (i.e.  to  teach  in  ways  that  support  21st 
century  learner  preferences),  21st  century  teaching  practices,  utilizing  methods  such  as 
cooperative  learning,  role  modeling,  etc.  were  not  consistently  observed  in  the  courses  included 
in  our  pilot  study.  Traditional  teaching  methods  utilizing  lecture,  PowerPoint  presentations,  etc. 
were  the  primary  methods  observed  (about  75  percent  of  the  time).  Hence,  near  term  research 
might  focus  on  better  understanding  the  reasons  for  this  misalignment  or  contradiction  to 
possibly  inform  changes  (using  current  “best”  practices)  to  the  structure  of  curriculum,  instructor 
training,  etc.  that  might  be  currently  constraining  instructors  in  the  classroom. 

The  technical  training  environment  offers  a  rich  opportunity  for  future  research  as 
demonstrated  by  the  challenges  cited  in  this  report  and  the  questions  and  topics  formulated  under 
each  of  the  six  research  vectors.  Since  numerous  technical  training  courses,  such  as  those 
observed  as  part  of  this  research  effort,  require  hands-on  involvement  (using  real  equipment  or 
trainers)  at  some  point  by  trainees  for  performance-based  training  evaluation,  it  will  be  important 
to  ensure  that  long  term  research  to  enhance  technical  training  is  focused  on  finding  the  right 
balance  or  mix  (in  terms  of  costs,  reduction  in  training  time,  and  learning  outcomes)  between  the 
use  of  advanced  training  technologies  and  methods  for  training,  and  traditional,  yet  highly 


27 


effective,  hands-on  training  methods.  This  is  probably  one  of  the  most  significant  considerations 
for  future  research  in  technical  training. 
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APPENDIX  A  -  Workshop  #  1  Agenda 


Anticipating  Future  Job-Aiding  /  Training  Requirements,  TACS  DO-9 

Workshop  #1  Agenda 
30-31  January  2008 
Sheppard  AFB,  TX 

Sheppard  AFB  Billeting  Office,  Conference  Room  #2  (Upstairs) 


30  January  2008  (Wednesday) 


Time 

Session 

PO  C/Format 

0730-0800 

Continental  Breakfast  (Conference  Room) 

Ms.  Gina  Johnson 

Mr.  Pat  Vincent 

0800-0815 

Welcome  /  Introductions  /  Admin  Remarks 

Mr.  Alex  Nelson 

Ms.  Gina  Johnson 

0815-0830 

Keynote  Address  (Col  Marti  Rossi,  82 
TRW/DS)  Director,  Wing  Staff 

Ms.  Gina  Johnson 
82TRW/TOO 

0830-0900 

82  TRW  Mission  Overview  and  Training 
Technology  function  (TOO-T)  Overview 

Presentation:  Ms.  Gina 
Johnson,  82TRW/TOO 

0900-0945 

TACS  DO-9  Overview  and  Discussion 

•  Task  Scope  and  Technical  Objectives 

•  Technical  Approach 

•  Accomplishments  to  Date 

•  Operational/Training  Environment 
(Challenges) 

•  Workshop  Objectives 

Presentation  (20-25  min): 
Mr.  Pat  Vincent 

Northrop  Grumman 

Corp. 

Ms.  Jill  Ritter 
AFRL/RHAL 

Discussion  (35-40  min) 

0945-1000 

Break 

1000-1045 

Future  of  AETC/Air  Force  Education  and 
Training 

Objective:  Provide  a  general  overview  of  the 
future  direction  of  AETC/AF  training.  Mr. 
Clemons  to  leave  for  airport  at  1045. 

Presentation  (15-20  min): 
Mr.  Larry  Clemons 

HQ  AETC/A5TT 

Discussion  (25-30  min) 

1045-1100 

Short  Presentations  by  Dr.  Lindsey/Dr.  Badler 

(5-10  min  each) 

1100-1215 

Lunch  at  Sheppard  AFB  Officer’s  Club 

This  will  be  a  cafeteria  style  lunch  and  Gina 
will  plan  to  invite  a  few  other  personnel  from 
the  82  TRW 

Ms.  Gina  Johnson 

1215-1315 

Training  /  Field  Evaluation  Process  and 
Feedback 

Presentation  (15-20  min): 
82  TRW  Training 
Evaluation,  Mr.  Gerald 
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Time 

Session 

PO  C/Format 

Objective:  Provide  an  overview  of  the 
training  evaluation  process,  types  and 
purpose  of  surveys  /  questionnaires  used  for 
evaluation.  Discuss  common  themes  across 
training  courses  that  point  to  current  training 
challenges  /  shortfalls.  How  do  career  fields 
compare  in  this  regard?  Are  some  courses 
more  problematic  in  terms  of  washback  rates, 
ability  to  train  certain  skills,  etc. 

Can- 

Discussion  (40-45  min): 
to  include  Mr.  Carr  and  a 
representative  from  each 
of  the  four  Group 

Training  Evaluation 
functions:  Gary 

Stevenson,  MSgt  Perez, 
Mike  Mauldwin,  Bobbie 
Merrifield 

1315-1430 

General  Overview  of  the  Instructional 

Systems  Development  (ISD)  Process 

Objective:  Provide  a  general  overview  of  the 
ISD  process  with  an  emphasis  on  training 
system  design  and  development.  Discussion 
on  how  well  does  the  current  ISD  process 
accommodate  future  training  or  learning 
environments?  What  are  the  challenges  ahead 
for  designing  and  developing  training  systems 
for  future  AF  learners? 

Presentation  (15-20  min): 
82  TRW  Faculty 
Development,  Presented 
by:  Cheryl  Whelan, 
Faculty  Development, 
Training  Development 
Chief 

Attended  by:  Barbara 
Fight,  Don  Williams,  and 
Beth  Yates,  Course 

Single  Point  Managers 
Discussion  (40-45  min) 
Group  Discussion 

1430-1445 

Break 

1445-1545 

Structured  Interview  with  Instructors 

Interviews  to  gain  insight  into  current  training 
challenges,  shortfalls,  and  perspectives  on 
alternative  approaches  to  designing  and 
delivering  training. 

AFRF  Research  Team, 

Ms.  Gina  Johnson,  Mr. 
Steve  Brite,  and  3-5 

82  TRW  Instructor 
Personnel  representing 
different  courses 

1545-1630 

Discussion  of  Training  Challenges  /  Shortfalls 

The  intent  of  this  session  would  be  to  begin  to 
develop  some  common  ground  / 
understanding  (based  on  the  material  and 
information  presented  earlier  in  the  day,  as 
well  as  interviews)  on  some  of  the  key 
challenges  and  shortfalls  related  to  training 
future  warfighters 

Working  Group 

Discussion 

AFRF  Research  Team, 

Ms.  Gina  Johnson,  and 

Mr.  Steve  Brite 

1630-1645 

Hot  Wash 

Mr.  Pat  Vincent 

Recap  highlights  and  review  actions  from  the 
day,  and  review  agenda  items  for  next  day 
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31  January  2008  (Thursday) 


Time 

Session 

POC/OPR 

0700-0730 

Coffee  and  Donuts  (Conference  Room) 

Ms.  Gina  Johnson 

0730 

Board  bus  and  travel  to  view  first  training  event 

Ms.  Gina  Johnson 

0730-0930 

Observe  Training  Events 

•  Block  7,  Electro-Environmental  Course  (high 
washback  rate  /  difficult  to  train) 

•  Fuels  Systems  Course  (technology  insertion  / 
use  of  equipment  in  classroom) 

•  Additional  Event  -  TBD 

Ms.  Gina  Johnson 

0930-1030 

Instructional  Technology  Units  (2),  and  Trainer 
Development  Round  table  in  Conference  Room 

Ms.  Gina  Johnson 

1030-1130 

Visit  with  BG  Devereaux,  82  TRW/CC 

Ms.  Gina  Johnson 

1130-1230 

Lunch  -  Golf  Course 

Ms.  Gina  Johnson 

1230 

Workshop  General  Session  Ends 
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APPENDIX  B  -  Instructor  Interview  Session  Questions 


Anticipating  Future  Job-Aiding  /  Training  Requirements  Workshop  #1 


1.  What  course(s)  do  you  currently  instruct? 

2.  How  long  have  you  been  an  Instructor/Instructor  Supervisor? 

3.  What  are  the  most  significant  training  challenges  you  are  up  against  in  providing  training  to 
your  students?  How  are  you  attempting  to  address  these  challenges? 

4.  Based  on  course/block  objectives  and  student  feedback/performance,  how  effective  do  you 
feel  the  current  methods  are  for  training  students  that  are  coming  through  your  courses? 

5.  What  training  methods  (in  ref.  to  the  mode  of  instruction)  do  you  think  work  best?  Which 
training  methods  do  you  feel  are  less  successful?  Why? 

6.  What  do  you  think  the  students  you  train  like  and  dislike  most  about  their  training? 

7.  Do  you  notice  a  trend  in  how  your  students  tend  to  learn  or  perform? 
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APPENDIX  C  -  Summary  of  82nd  CoP  Responses  to  Instructor  Interview  Session  Questions 

Responses  to  Questions  #1  and  #2  pertained  to  the  specific  course  an  instructor  was  assigned  and 
their  years  of  experience.  This  data  is  not  included  in  the  report. 

Question  #3  -  What  are  the  most  significant  training  challenges  you  are  up  against  in 
providing  training  to  your  students?  How  are  you  attempting  to  address  these  challenges? 

■  Shortage  of  Instructor  personnel  (manning) 

■  Instructor  workload  is  significant  -  course  related  and  additional  duties 

■  Outdated  and  broken  training  equipment  /  trainers  /  tools 

■  Aircraft  -  age  and  availability 

■  Lack  of  funding  to  purchase  equipment  /  tools  /  computers 

■  Technology 

■  Keeping  pace  with  new  technologies  and  insertion  into  training 

■  Lag  time  in  incorporating  technology  due  to  funding  /  resource  constraints 

■  Keeping  students  “engaged”  in  course  material  /  training 

■  Motivation  of  students  to  learn  /  self-discipline 

Question  4  -  Based  on  course/block  objectives  and  student  feedback  /  performance,  how 
effective  do  you  feel  the  current  methods  are  for  training  students  that  are  coming  through 
your  courses? 

■  General  consensus  was  that  current  methods  for  training  students  were  “good”  / 
“effective” 

■  A  few  indicated  that  some  course  material  was  outdated  /  not  relevant 

■  Some  indicated  a  need  to  introduce  more  technology  (beyond  Powerpoint  presentations) 
-  primarily  to  engage  students  and  reinforce  material 

Question  5  -  What  training  methods  (in  ref.  to  the  mode  of  instruction)  do  you  think  work 
best?  Which  training  methods  do  you  feel  are  less  successful?  Why? 

■  Hands-on  training  is  best  (maintenance  courses  in  particular) 

■  Lecture  and  PowerPoint  presentations  are  not  that  effective  or  interesting  to  the  students 

■  Modeling  and  simulation  for  demonstration  or  practice  would  help  reinforce  some 
concepts  and  reduce  wear  and  tear  on  trainers  /  equipment 
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Question  6  -  What  do  you  think  the  students  you  train  like  (+)  and  dislike  (-)  most  about 
their  training? 


■  +  Hands-on  /  interactive  training  and  demonstrations 

■  -  Lecture  /  Powerpoint  presentations  and  “knowledge”  objectives 

■  -  Military  training  activities  and  additional  duties  outside  the  classroom 

■  -  Time  it  takes  in  training  to  get  their  hands  on  the  actual  aircraft  or  equipment 

■  -  Fast  pace  of  instruction  in  some  courses 

■  -  Downtime  during  training  (particularly  during  class  time  because  of  lack  of  equipment, 
instructors,  etc.) 

■  -  Limited  or  no  access  to  course  training  materials  outside  of  class  for  study  purposes 

Question  7  -  Do  you  notice  a  trend  in  how  your  students  tend  to  learn  or  perform? 

■  Majority  indicated  that  they  did  seem  to  notice  a  trend  in  how  their  students  learned  / 
performed  -  only  some  qualified  their  responses 

■  About  15  percent  indicated  they  did  not  notice  a  trend 

■  Some  of  the  comments  pertaining  to  trends  included: 

■  Younger  students  seem  to  respond  better  interactive  technology  and  computer 
based  applications 

■  Students  seem  to  learn  quicker  when  they  can  move  from  the  classroom  to  the 
hands-on  environment 

■  Some  students  seem  more  lethargic  because  of  the  length  of  day  that  includes 
course  time  and  military  training  /  additional  duties 

■  Some  students  have  a  problem  with  time  management 


36 


APPENDIX  D  -  21st  Century  Training  for  21st  Century  Learners  -  A  Pilot  Research  Study 


Jill  L.  Lindsey,  Ph.D. 
Dayton,  Ohio 
December  2008 


Accomplished  as  part  of  the  research  conducted  for  TACS  DO-9 
Funded  by  AFRL/RHAL 


37 


Executive  Summary 


A  pilot  research  project  was  undertaken  to  better  understand  the  learning  preferences  of  Air 
Force  technical  trainees  so  as  to  recommend  modifications  and  redesign  of  technical  training  to 
improve  engagement,  comprehension,  and  successful  course  completion.  The  goals  of  the  study 
were: 


1.  Determine  whether  current  Air  Force  technical  trainees’  are  21st  century 
learners. 

2.  Assess  whether  Air  Force  technical  training  is  aligned  with  trainees’  learning 
preferences  and  identify  areas  of  potential  improvement. 

3.  Identify  leverage  points  in  training  where  innovations  in  technology  could 
potentially  improve  training  outcomes. 

To  accomplish  these  goals,  a  review  of  literature  was  conducted,  an  observation  rubric  was 
developed  for  use  in  technical  training  classroom  observations,  and  responses  on  preference 
inventories  were  collected  from  trainees  and  instructors. 

The  review  of  literature  revealed  a  growing  consensus  regarding  the  attributes  and  preferences  of 
21st  century  learners  and  recommended  instructional  practices  to  best  meet  their  needs.  The 
literature  suggested  that  21st  century  learners: 

a.  desire  meaningful  work 

b.  are  self-directed  in  setting  and  achieving  goals 

c.  prefer  working  collaboratively 

d.  have  a  sense  of  capability  or  competence 

e.  have  strong  visual-spatial  skills 

f.  tend  toward  parallel  processing  (multi-tasking) 

g.  are  technologically  intuitive 

The  recommended  instructional  methods: 

a.  provide  opportunities  for  collaboration  using  Cooperative  Learning 

b.  harness  learners’  technology  authority  and  innovation 

c.  recognize  their  visual-spatial  skills  and  ability  to  parallel  process 

d.  require  them  to  take  responsibility  for  their  own  learning  by  providing  opportunities 
for  choices,  decision-making,  and  selecting  learning  strategies 

e.  give  learners  autonomy  while  establishing  high  expectations  and  encouraging 
metacognition 

f.  support  learners’  sense  of  competence  with  frequent  praise  and  feedback 

g.  assign  meaningful  work 

The  Grasha-Reichmann  Learning  Styles  Inventory  (LSI)  and  Teaching  Styles  Inventory  (TSI) 
were  selected  to  identify  the  learning  preferences  of  trainees  and  the  teaching  preferences  of 
instructors.  Grasha’s  research  identified  four  common  clusters  of  teaching  styles  and  associated 
student  learning  styles  for  the  most  effective  teaching-learning  dynamic.  Each  of  the  Teaching 
Styles  Clusters  and  learning  styles  composites  are  made  up  of  two  or  three  primary  styles  with 
one  or  more  secondary  styles.  The  learning  and  teaching  styles  identified  by  the  LSI  and  TSI 
correlated  but  did  not  directly  align  with  the  21st  century  preferences  and  methods  identified  in 
the  literature,  and  no  items  addressed  technology  use.  Therefore,  ten  additional  items  related  to 
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technology  use  were  added  to  the  LSI  and  TSI.  Selected  items  from  the  LSI  were  also  mapped  to 
six  21st  century  learning  preferences  to  enable  the  computation  of  21st  century  learning 
preference  scores.  To  measure  the  degree  of  alignment  between  21st  century  learning 
recommendations  and  Air  Force  technical  training  methods,  a  21st  Century  Technical  Training 
Observation  Rubric  (TTOR)  was  developed.  The  rubric  identified  nine  criteria  with  three  or 
more  examples  of  possible  evidence.  The  first  eight  criteria  described  21st  century  instruction  and 
the  ninth  criteria  documented  traditional  instructional  methods. 

The  TTOR  was  used  to  document  classroom  observations  of  seventeen  instructors  in  six  courses. 
The  observation  protocol  involved  teams  observing  each  class  for  20-30  minutes  with  note¬ 
taking,  followed  by  assigning  ratings  based  on  the  evidence  noted.  Observers  individually 
completed  their  ratings  for  each  TTOR  criteria  (0=  not  observed,  1=  seldom  observed,  2= 
observed  about  half  of  the  time,  3=  observed  most  or  all  of  the  time).  Following  each  observation 
period,  the  observers  gathered  to  discuss  their  evidence  and  individual  ratings  for  each 
class/instructor.  Most  observations  resulted  in  common  ratings  by  the  observers  but  when 
differences  in  individual  ratings  occurred,  discussion  of  evidence  resulted  in  a  consensus  rating. 
Observed  instruction  was  dominantly  traditional  using  lectures  with  slides,  but  a  few  classes 
included  active  student  participation  in  21st  century  methods. 

The  Learning  Styles  Inventory  (LSI)  was  administered  to  172  trainees.  Responses  on  the  LSI 
supported  the  generalization  that  AF  technical  trainees  are  21st  century  learners  with  a  mean  LSI 
composite  of  Participant/Dependent/Collaborative.  The  Teaching  Styles  Inventory  (TSI)  was 
administered  to  22  technical  training  instructors  in  six  courses.  Instructor  responses  indicated 
they  preferred  Teaching  Style  Cluster  2-  Personal  Model/Expert/Formal  Authority  with 
secondary  preferences  for  the  Facilitator/Delegator  styles.  Grasha’s  research  suggests  that 
Teaching  Styles  Cluster  2  best  supports  students  with  the  Participant/Dependent/Collaborative 
learning  styles  composite. 

The  LSI-TSI  data  indicated  there  was  alignment  between  instructors’  preferred  teaching  cluster 
and  trainees’  learning  preferences.  However,  responses  on  the  technology  use  items  indicated 
that  trainees  preferred  more  technology  use  in  instruction  than  did  their  instructors.  In  addition, 
while  the  LSI  and  TSI  data  indicated  alignment  between  instructors’  and  trainees’  preferences, 
classroom  observation  ratings  using  the  TTOR  indicated  instructors  were  not  observed  teaching 
in  their  preferred  Teaching  Styles  Cluster.  Observed  instruction  was  not  representative  of  21st 
century  methods. 

The  findings  of  this  pilot  study  suggested  that  technical  training  instruction  could  be  better 
aligned  to  meet  both  instructors’  preferred  teaching  styles  and  trainees’  learning  preferences. 
Adjustments  in  teaching  to  include  more  21st  century  methods  would  accommodate  both 
instructors’  and  trainees’  preferences.  Trainees  expressed  the  desire  for  more  technology  use  in 
the  classroom  and  for  digital  access  to  course  materials.  Even  though  instructors’  responses  on 
technology  items  were  slightly  lower  than  those  of  trainees,  instructors’  ratings  indicated  they 
would  like  to  incorporate  more  technology  into  the  training  process.  A  variety  of  technologies 
beyond  the  common  power-point  lecture  are  available  to  incorporate  into  the  learning 
environment.  Online  wikis  and  blogs  can  also  help  build  collaboration  and  communications 
skills,  while  social  networks  provide  access  to  expensive  or  scarce  high-level  tools.  When  guided 
by  training  objectives,  small-scale  shifts  in  technology  use  can  offer  trainees  meaningful  learning 
experiences  and  increased  interactivity.  This  is  an  area  where  technology  innovations  could 
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improve  engagement  and  learning  outcomes.  Further  research  is  needed  to  determine  the  impact 
of  electronic  course  materials  and  digital  communications  in  technical  training. 

The  alignment  between  instructors’  and  trainees’  preferences  is  good  news,  in  that  it  suggests 
that  instructors  possess  the  desire  and  inclination  to  teach  in  ways  that  support  21st  century 
learners’  preferences  for  participation,  self-direction,  competence  and  collaboration.  However, 
observed  instruction  did  not  utilize  recommended  21st  century  practices  such  as:  cooperative 
learning,  role  modeling,  illustrations,  problem-solving,  multiple  examples,  discussions  of 
alternative  approaches,  sharing  of  thought  processes  for  obtaining  answers,  sharing  of  personal 
experiences,  coaching  and  guiding  with  feedback,  or  visual  tools.  Since  instructors  preferred  but 
were  not  observed  enacting  21st  century  teaching  methods  in  the  classroom,  there  is  need  for 
further  exploration  of  this  contradiction.  Perhaps  aspects  of  the  training  environment  or  the 
structure  of  the  curriculum  are  constraining  instructors  from  teaching  in  their  preferred  styles. 
Perhaps  Instructor  Training  should  be  redesigned  to  prepare  instructors  to  use  21st  century  best 
practices.  These  are  a  few  of  the  many  avenues  for  future  research  revealed  by  this  study.  The 
evidence  from  this  pilot  study  revealed  that  Air  Force  technical  trainees  are  21st  century  learners 
and  improving  technical  training  instruction  through  the  use  of  21st  century  practices  merits 
further  investigation. 
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Introduction 


There  is  an  ever  present  need  to  improve  technical  training  so  as  to  efficiently  produce  qualified 
airmen  while  reducing  the  time  and  costs  associated  with  preparing  them  for  field-ready  status. 
Innovative  technology  can  play  a  role  in  such  improvements,  but  only  to  the  extent  that  the 
technology  actually  improves  training  practices  and  meets  learners’  needs.  Before  making 
investments  in  technology,  the  Air  Force  must  examine  the  technical  training  environment  in  the 
context  of  today’s  learners  in  order  to  identify  those  places  where  innovation  will  offer  the 
greatest  impact.  At  the  most  fundamental  level,  the  trainee’s  experience  and  needs  as  a  learner 
must  be  considered.  A  recent  report  on  Human  Behavior  in  Military  Contexts  (Blascovich  & 
Hartel,  2008)  concluded  that  “first  and  foremost  training  programs  should  be  based  on  an 
understanding  of  how  people  learn  and  how  instructional  methods  affect  learning.” 

The  learners  in  the  21st  century  are  different  in  many  ways  from  their  instructors.  There  is  a 
growing  body  of  literature  describing  the  21st  century  learner  as  interactive,  collaborative, 
experimental,  multi-tasking,  visual  and  kinesthetic  (Oblinger  &  Oblinger,  2005).  A  better 
understanding  of  the  learning  needs  of  21st  century  trainees  can  guide  the  modification  and 
redesign  of  technical  training  instruction,  methods,  policies,  and  practices  to  improve  student 
engagement,  comprehension,  and  successful  course  completion.  It  can  also  provide  decision 
makers  evidence-based  recommendations  for  selecting  technologies  with  the  highest  potential  to 
improve  learning  outcomes  and  graduation  rates.  To  this  end,  a  pilot  research  project  was 
undertaken  with  the  following  goals: 

•  To  better  understand  current  technical  training  challenges  related  to  trainees’ 
attributes  and  learning  preferences 

•  To  examine  alignment  of  teaching  and  learning  modalities  in  current  technical 
training  and  identify  areas  for  improvement 

•  To  identify  leverage  points  in  training  where  innovations  in  technology  could 
improve  training  outcomes 

The  project  unfolded  in  four  phases: 

1.  A  review  of  literature  exploring  the  evolving  consensus  among  researchers  regarding  21st 
century  learners’  attributes  and  preferences  (predispositions,  skills,  and  preferred  learning 
modalities),  and  identified  “best  practices”  in  training  and  education  to  meet  the  needs  of  21st 
century  learners. 

2.  Data  Gathering  Tools. 

a.  The  development  of  a  21st  Century  Technical  Training  Observation  Rubric  for  use  in 
assessing  the  degree  of  alignment  between  recommended  21st  century  training  practices  and 
current  technical  training  practices. 

b.  The  selection  of  the  Grasha-Reichmann  Learning  Styles  Inventory  (LSI)  to  administer 
to  a  sample  of  trainees  to  determine  if  their  learning  preferences  match  preferences  described  in 
the  literature  about  21st  century  learners. 

c.  The  selection  of  the  Grasha-Reichmann  Teaching  Styles  Inventory  (TSI)  to  administer 
to  technical  training  instructors  to  identify  their  teaching  styles  preferences. 
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3.  Data  Collection  and  Analysis. 

a.  Conducted  observations  (n=17)  in  six  technical  training  courses  using  the  21st  century 
Technical  Training  Observation  Rubric. 

b.  Collected  Learning  Styles  Inventory  responses  from  a  sample  of  technical  trainees. 

c.  Collected  Teaching  Styles  Inventory  responses  from  a  sample  of  technical  training 
instructors. 

4.  Discussion  of  the  findings  and  recommendations  related  to  improving  technical  training. 

This  paper  describes  the  activities  and  insights  gained  in  each  of  these  phases. 

1.  Review  of  Literature 

A  review  of  literature  was  undertaken  to  explore  articles,  texts  and  research  studies  related  to  21st 
century  learners’  attributes  and  to  identify  instructional  practices  designed  to  best  meet  the  needs 
of  21st  century  learners.  The  umbrella  of  learner  attributes  explored  included  predispositions, 
types  of  knowledge,  identified  strengths  and  skills,  and  preferred  learning  modalities.  The  review 
of  literature  revealed  a  growing  consensus  regarding  the  attributes  and  preferences  of  21st 
century  learners  and  recommended  instructional  practices  to  best  meet  their  needs. 

Who  are  21st  Century  learners?  Twenty-first  century  learners  are  considered  “Millennials”  born 
in  or  after  1980,  and  make  up  nearly  30  percent  of  the  American  population.  They  are  the  most 
racially  and  ethnically  diverse  generation  to  have  come  along  in  American  history.  A  variety  of 
labels  have  been  applied  to  the  sub-groups  within  the  Millennial  umbrella,  the  most  common  are 
Generations  Y  and  Z.  Those  who  subscribe  to  the  alphabetical  subdivisions  separate  the  groups 
by  birth  years:  members  of  Generation  Y  were  born  between  1980-1994  and  Generation  Z 
between  1995-2009.  Those  who  distinguish  between  these  two  groups  do  so  because  Generation 
Z  is  the  world’s  first  generation  to  be  considered  digital  “natives,”  where  Generation  Y  grew  up 
during  the  development  period  of  computer  technologies  and  the  Internet.  There  are  many 
popular  labels  for  21st  century  learners:  dot  com  kids,  the  Net  Generation,  Generation  Media,  and 
Generation  C  (with  the  "C"  meaning  click,  connected,  community,  and/or  creative). 

21st  century  learners  or  Millennials,  like  other  generations,  are  shaped  by  the  events, 
developments,  and  trends  of  their  time  (Howe  &  Strauss,  1991).  According  to  generational 
research  (Howe  &  Strauss,  1991),  many  parallels  exist  between  Millennials  and  the  World  War 
II  generation.  Both  groups  experienced  the  building  of  new  schools,  curriculum  focused  on  math 
and  science,  standardized  testing,  character  education  in  schools,  school  uniforms,  youth 
volunteerism,  government  protections,  and  the  popularity  of  the  Superman  archetype.  They  also 
hold  similar  values:  teamwork,  a  sense  of  capability  or  competence,  community  before  self,  and 
respect  for  parents.  These  values  appear  to  contribute  to  and  support  the  decision  to  volunteer  for 
community  and  military  service. 

Beyond  these  parallels,  there  are  vast  differences  that  make  Millennials  a  unique  generation. 
Millennials  are  viewed  as  the  connected  generation  because  they  have  had  lifelong  use  of 
communications,  the  Internet,  and  media  technologies.  In  America  and  Europe,  they  have  also 
grown  up  in  a  world  with  widespread  equality  of  the  sexes  and  single -parent  or  same-sex  parent 
families.  Distinguishing  differences  between  Generation  Y  and  Z  are  their  family  contexts  and 
political  climates.  Generation  Y  was  the  “two-income,  daycare”  generation  while  Generation  Z  is 
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the  first  generation  after  the  Baby  Boomers  to  be  raised  at  home  by  parents  and  a  single  income 
family.  Generation  Y  grew  up  in  a  post-Cold  War  environment,  but  Generation  Z's  landscape  is 
dominated  by  the  after  effects  of  9/1 1. 

What  are  the  key  attributes  and  strengths  of  the  21st  century  learner?  A  growing  body  of 
literature  suggests  that  21st  century  learners  can  be  described  using  a  constellation  of  key 
attributes  and  preferences  (McGlynn,  2005,  2008;  Montana  &  Petit,  2008;  Vander  Schee,  2008; 
Abram,  2007;  Moore,  2007;  Dordai  &  Rozzo,  2006;  Martin  &  Tulgan,  2006;  Skiba  &  Barton, 
2006;  Johnson  &  Romanello,  2005;  Oblinger,  2003;  Raines,  2003:  Ben-Jacob,  Levin  &  Ben- 
Jacob,  2000;  Howe  &  Straus,  2000).  They:  a)  desire  meaningful  work,  b)  are  self-directed 
in  setting  and  achieving  goals,  c)  prefer  working  collaboratively,  d)  have  a  sense  of  capability  or 
competence,  e)  have  strong  visual-spatial  skills,  f)  tend  toward  parallel  processing  (multi¬ 
tasking),  and  g)  are  technologically  intuitive.  While  authors  used  a  variety  of  labels  and 
descriptors  for  21st  century  learners,  these  key  preferences  emerged  as  common  inter-related 
themes. 

Most  authors’  descriptions  focused  on  the  interactive  nature  of  21st  century  learners,  linking  the 
desire  for  social  interaction  and  collaboration  with  the  use  of  technology.  This  common  cluster 
of  attributes  indicates  21st  century  learners  are  digitally  literate,  mobile,  experiential,  social,  and 
crave  interactivity  and  immediacy  (Skiba  &  Barton,  2006;  Oblinger,  2003).  They  prefer  to  learn 
by  working  collaboratively  and  through  teamwork  (Koster  &  Smith,  2007;  Rodgers,  Runyon, 
Starrett,  &  VonHolzen,  2006;  Skiba  &  Barton,  2006;  Johnson  &  Romanello,  2005;  Oblinger, 
2003).  Millennial  are  fluent  in  the  digital  language  of  computers,  video  games,  and  the  internet 
(Prensky,  2001).  Having  been  socialized  in  a  digital  world,  they  are  more  technically  literate  and 
connected  to  digitally  streaming  information  than  previous  generations  (Eisner,  2005).  Some 
posit  the  increased  need  for  socialization  is  actually  a  by-product  of  intense  exposure  to  a  wide 
array  of  communication  technologies  (Oblinger,  2003).  Millennial  not  only  use  technology 
heavily,  but  also  trust  it  implicitly  (Hartman,  Moskal,  &  Dziuban,  2005).  This  mobile  generation 
uses  PDA’s,  cell  phones,  MP3  players,  and  lap-tops  to  stay  connected  throughout  the  day. 
Utilizing  different  technologies,  the  Millennial  is  able  to  set  up  a  connected  world  without 
having  to  stay  in  a  fixed  location.  Millennials  are  comfortable  with  and  enjoy  using  technology 
(McGlynn,  2005)  and  prefer  to  use  it  whenever  possible  (Abram,  2007).  Often  it  seems  as  though 
the  ability  to  use  a  variety  of  technological  devices  is  almost  intuitive  (Oblinger  &  Oblinger, 
2005). 

A  survey  of  275,000  high  school  students  identified  volunteerism,  collaboration  and  technology 
use  as  common  behaviors  and  preferences  (Oblinger,  2003). 

•  74  percent  volunteered  in  the  last  year/  65  percent  in  the  last  month 

•  94  percent  used  the  Internet  for  school  work 

•  78  percent  indicated  the  Internet  helped  them  with  their  school  work 

•  81  percent  used  E-mail  to  talk  with  friends 

•  56  percent  preferred  E-mail  to  the  telephone 

•  56  percent  said  the  Internet  improved  relationships  with  teachers 

•  89  percent  received  some  information  from  their  teachers  via  the  Internet 

•  19  percent  would  rather  use  E-mail  than  have  a  face  to  face  interaction  with  a  teacher 
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While  instant  communication  technologies  may  explain  the  proclivity  for  peer  interactions  and 
immediacy,  a  related  attribute  is  parallel  processing.  21st  century  learners  like  to  parallel  process 
or  multi-task,  shifting  their  attention  rapidly  from  one  activity  to  another,  and  may  choose  not  to 
pay  attention  to  things  that  don’t  hold  their  interest  (Oblinger  &  Oblinger,  2005).  Parallel 
processing  is  a  way  of  life  for  the  Millennial,  but  some  (Abram,  2007)  worry  that  the 
multitasking  lifestyle  of  Millennials  contributes  to  a  lack  of  focus  and  poor  concentration  skills. 
Rodgers,  Runyan,  Starrett,  &  Van  Holsen  (2006)  found  Millennials  are  good  at  reading  visual 
images  but  weak  in  reading  skills,  have  strong  visual-spatial  skills,  tend  toward  parallel 
processing  and  inductive  discovery,  and  have  fast  response  time  which  tends  to  equal  shorter 
attention  spans.  Increases  in  parallel  processing  may  contribute  to  shorter  attention  spans,  but 
Millennials  appear  quite  comfortable  when  engaged  in  multiple  activities  simultaneously 
(Oblinger,  2003).  Through  a  variety  of  technologies,  Millennials  are  able  to  incorporate 
multitasking  into  day-to-day  activities  and  satisfy  their  high  need  for  interconnectivity  (Hartman, 
Moskal,  &  Dziuban,  2005). 

Another  cluster  of  related  attributes  in  the  literature  were  competence,  self-direction,  and 
meaningful  work.  In  a  consumer  trend  research  report  for  Iconoculture,  Koster  &  Smith  (2007) 
described  four  major  characteristics  of  Millennials:  collaboration,  authenticity,  flexibility,  and 
reciprocity.  These  later  three  attributes  work  together  to  create  a  sense  of  competence  in  the  21st 
century  learner. 

They're  natural  team  players  with  an  innate  sense  of  entitlement  that  tells  them 
that  the  entry-level  sales  rep  deserves  to  have  just  as  much  input  as  the  senior 
manager...  They  prefer  flexibility,  freedom,  independence,  and  control... Life  is  a 
transaction. .  ."you  get  what  you  give"  is  the  golden  rule. . .  Reciprocity  is  very 
important  when  you  are  raised  to  believe  that  you  always  have  a  choice  and  you 
can  always  expect  someone  to  want  to  hear  your  point  of  view.  Connecting  with 
this  demographic  means  accommodating  their  need  to  be  heard  and  be  counted. 

The  need  for  recognition  is  integral  to  their  sense  of  competence,  in  that  Millennials  rely  on 
feedback  and  recognition  to  support  and  nurture  feelings  of  competence.  A  recent  study 
(Montana  &  Petit,  2008)  explored  the  motivators  for  Generations  X  and  Y  and  found  the 
strongest  motivators  for  both  groups  were  recognition,  followed  by  high  income  or  good  pay, 
good  work  relations,  career  growth  opportunities,  and  the  chance  to  do  interesting/meaningful 
work. 

This  generation  is  very  self-reliant  and  feels  a  need  to  achieve  (Eisner,  2005).  This  need  to 
achieve  often  harmonizes  with  the  Millennials  demand  for  meaningful  work.  The  Center  for 
Information  Research  on  Civic  Learning  &  Engagement  (2007)  has  recently  published  research 
on  the  many  ways  Millennials  are  more  actively  involved  in  their  communities.  The  study 
compared  Millennials  with  Boomers  and  found  Millennials  have  a  great  deal  more  experience 
with  volunteering  and  believe  in  their  obligation  to  work  together  with  others  on  important  social 
issues.  This  need  for  meaningful  work  is  not  limited  to  volunteering  and  social  action.  In  a 
Northwestern  Mutual  (2004)  study  of  desirable  job  traits,  nearly  half  of  all  respondents  ranked 
“doing  work  which  gives  me  the  opportunity  to  help  others”  as  very  important.  Assigning 
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meaningful  work  and  demonstrating  the  relevance  of  information  are  essential  components  in 
engaging  21st  century  learners  (McGlynn,  2008). 

Meaningful  work  is  related  to  competence  and  self-direction,  in  that  Millennials  need  superiors 
and  peers  to  support  them  and  nurture  their  growing  competence  with  immediate,  constructive 
feedback  (Martin  &  Tulgan,  2006;  Johnson  &  Romanello,  2005;  Wagner,  2005).  Self-direction  is 
the  third  key  attribute  in  this  cluster.  21st  century  learners  are  independent,  entrepreneurial 
thinkers,  who  love  freedom  and  flexibility,  and  like  having  a  sense  of  choice  (Martin  &  Tulgan, 
2006).  When  working  on  projects,  the  Millennial  is  likely  to  explore  multiple  options  before 
deciding  on  a  set  path  or  focus  (Abram,  2007). 

This  review  of  literature  converged  around  several  key  attributes  of  21st  century  learners: 
meaningful  work,  self-direction,  competence,  collaboration,  technology  use,  and  parallel 
processing.  Once  identified,  these  attributes  were  used  to  help  guide  the  review  of  literature 
identifying  best  instructional  practices  for  21st  century  learners. 

What  are  best  practices  for  teaching  and  training  21st  century  learners?  Millennials  are  highly 
collaborative  and  self-directed  but  in  need  of  feedback  and  support  to  nurture  their  sense  of 
competence.  They  learn  in  a  variety  of  ways  while  parallel  processing,  are  motivated  and 
engaged  by  meaningful  work,  and  are  technologically  literate.  There  exist  a  variety  of 
suggestions  for  how  to  optimize  these  attributes  and  preferences.  The  most  frequently 
recommended  instructional  methods  align  with  the  attributes  and  strengths  of  21st  century 
learners: 

a.  provide  opportunities  for  collaboration  using  Cooperative  Learning 

b.  harness  learners’  technology  authority  and  innovation 

c.  recognize  their  visual-spatial  skills  and  ability  to  parallel  process 

d.  require  them  to  take  responsibility  for  their  own  learning  by  providing  opportunities 

for  choices,  decision-making,  and  selecting  learning  strategies 

e.  give  learners  autonomy  while  establishing  high  expectations  and  encouraging 

metacognition 

f.  support  learners’  sense  of  competence  with  frequent  praise  and  feedback 

g.  assign  meaningful  work 

Learning  experiences  that  encourage  collaboration  optimize  21st  century  learners’  social  and 
interactivity  needs.  Millennials  enjoy  working  in  groups  and  collaboratively  constructing 
knowledge  within  a  social  community  (Skiba  &  Barton,  2006).  Cooperative  Learning  is  a  well- 
researched  teaching  method  with  results  that  demonstrate  students  learn  faster  and  more 
efficiently,  have  greater  retention,  and  feel  more  positive  about  their  learning  experiences.  A 
Johnson,  Johnson  &  Stanne  (2000)  meta-analysis  of  164  studies  investigating  eight  different 
types  of  cooperative  learning  methods  found  all  eight  cooperative  learning  methods  had  a  greater 
impact  on  student  achievement  when  compared  with  individualistic  and  competitive  learning 
methods.  Methods  of  Cooperative  Learning  create  relationships  between  group  members  based 
on  “positive  interdependence  (a  sense  of  sink  or  swim  together),  individual  accountability  (each 
member  has  to  contribute  and  learn),  interpersonal  skills  (communication,  trust,  leadership, 
decision  making,  and  conflict  resolution),  face-to-face  interaction,  and  reflecting  on  how  well  the 
team  is  functioning  and  how  to  function  even  better”  (Johnson  &  Johnson,  1997).  Thus 
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cooperative  learning  methods  not  only  address  21st  century  learners’  preference  for  collaboration, 
but  also  address  their  needs  for  self-direction,  metacognition,  and  competence. 

Millennials  have  set  themselves  apart  from  previous  generations  in  their  desire  to  actively 
construct  knowledge  through  interaction  (Skiba  &  Barton,  2006;  Oblinger  &  Oblinger,  2005). 
This  emphasis  on  social  learning  is  in  sharp  contrast  to  the  traditional  view  that  knowledge  is 
passed  down  from  expert  to  novice,  teacher  to  student.  This  does  not  mean  that  the  traditional 
classroom  lecture  has  to  be  abandoned.  Interactive  lectures  can  encourage  collaborative 
meaning-making  by  pairing  short  lectures  with  simulations,  group  activities,  interactive 
exercises,  and  game-style  review  sessions  (McGlynn,  2008;  Moore,  2007;  Johnson  & 

Romanello,  2005).  Technology  tools,  like  Tablet  PC’s  or  hand-held  PDAs,  could  also  be  used  to 
create  interactive  lectures  by  allowing  trainees  to  fill  in  a  class  outline  or  respond  to  key 
questions  as  the  lecture  progresses.  These  interactive  tools  allow  learners  to  share  ideas  and 
collectively  brainstorm  by  submitting  responses  the  entire  class  can  see  instantaneously. 

As  “digital  natives,”  21st  century  learners  have  spent  their  entire  lives  surrounded  by  the  toys  and 
tools  of  the  digital  age:  computers,  videogames,  digital  music  players,  DVD  players,  digital 
cameras,  and  cell  phones.  Technology  is  an  assumed  part  of  the  environment.  Millennials  expect 
instructors  to  use  technology  to  communicate,  and  many  consider  a  balanced  use  of  technology 
in  the  learning  environment  to  be  essential  (Herz,  2005).  A  variety  of  technologies  beyond  the 
common  power-point  lecture  are  available  to  incorporate  into  the  learning  environment.  Email, 
online  chat,  wikis  and  blogs  can  also  help  build  collaboration  and  communications  skills 
(Hartman,  Moskal  &  Dziuban,  2005).  Online  social  networks  such  as  Second  Life,  e-science  labs 
and  e-humanities  can  provide  access  to  expensive  or  scarce  high-level  tools.  When  guided  by 
training  objectives,  small-scale  shifts  in  technology  use  can  offer  trainees  meaningful  learning 
experiences  and  increased  interactivity. 

Since  21st  century  learners  have  strong  visual-spatial  skills,  incorporating  visual  tools  into 
teaching  and  learning  helps  students  process  and  remember  important  information.  “Visual  tools 
are  nonlinguistic  symbol  systems  that  graphically  link  mental  and  emotional  associations  to 
create  and  communicate  rich  patterns  of  thinking”  (Hyerle,  2009,  p  xix).  These  visual-spatial- 
verbal  displays  of  information  help  visual- spatial  thinkers  translate  information  into  personal 
knowledge.  They  are  also  metacognitive  tools  that  help  learners  reflect  upon  their  thinking. 
There  are  four  kinds  of  visual  tools:  brainstorming  webs,  graphic  organizers,  conceptual  maps, 
and  Thinking  Maps.  They  can  be  used  as  communication  tools  for  collaboration  and  networking 
or  to  help  an  individual  organize  their  thoughts.  Research  in  the  neurosciences  suggests  that  the 
human  brain  organizes  information  in  networks  and  maps,  and  the  brain  is  dominantly  visual 
receiving  about  70  percent  of  the  information  from  our  environment  through  our  eyes  (Wolfe, 
2004).  New  computer  and  digital  technologies  are  reinforcing  this  dominant  processing  ability. 
Unlike  linear-  auditory  lectures,  visual  tools  actively  engage  students  in  the  process  of  learning 
and  can  include  opportunities  for  parallel  processing  -  thinking  about  and  doing  more  than  one 
thing  at  a  time.  The  combination  of  visual  and  verbal  processing  through  mapping  allows  the 
human  brain  to  engage  in  dual  coding,  which  when  focused  on  specific  content  can  optimize 
human  learning  and  memory  (Marzano,  Pickering,  &  Pollock,  2001). 

Another  recommended  method  for  meeting  21st  century  learners’  needs  is  Problem  Based 
Learning  (PBL),  which  provides  learners  with  choices  and  decision-making  opportunities  that 
require  them  to  take  responsibility  for  their  own  learning,  engage  in  meaningful  work,  and  reflect 
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upon  their  learning.  PBL  encourages  learners  to  be  self-directed  by  constructing  new  knowledge 
from  problem  solving  experiences  (Brown,  2005).  In  PBL,  students  are  presented  with  a  problem 
they  attempt  to  solve  using  knowledge  and  reasoning  skills  they  already  possess.  Interactively, 
they  identify  the  knowledge  and  skills  they  are  lacking,  and  design  a  plan  for  meeting  their 
learning  needs.  After  seeking  the  new  information  or  practicing  needed  skills,  they  apply  their 
newly  acquired  knowledge  and  skills  to  the  problem,  and  describe  what  they  have  learned  (Ben- 
Jacob,  Levin,  &  Ben-Jacob,  2000).  An  important  part  of  this  cycle  is  identifying  knowledge 
deficiencies  relative  to  the  problem;  these  knowledge  deficiencies  become  the  learning  issues 
that  students  need  to  research  in  order  to  solve  the  problem  (Hmelo-Silver,  2004).  Students  can 
then  utilize  different  avenues  to  research  information;  texts  and  written  materials,  online  e- 
laboratories,  publications  databases,  films,  documentaries,  or  internet  connections  across  the 
globe  with  experts  in  the  field.  PBL  helps  trainees  become  active  constructors  of  knowledge, 
knowledge  that  is  learned  in  meaningful  contexts  that  are  similar  to  those  in  which  they  will 
work  (Artino,  2008). 

Incorporating  metacognition,  asking  students  to  reflect  upon  their  learning,  is  an  important 
component  of  effective  teaching  for  21st  century  learners.  Metacognition  engages  learners  by 
getting  them  to  analyze  their  own  learning  strategies  and  make  decisions  about  what  kinds  of 
learning  experiences  and  information  they  still  need  to  fully  understand  the  material. 
Neuroscience  research  on  human  learning  has  found  that  synaptic  connections  are  reinforced  and 
strengthened  by  self-monitoring  metacognitive  activities,  improving  memory  and  efficient 
retrieval  of  information  (Bradsford,  Brown  &  Cocking,  1999).  Metacognitive  modeling  by 
instructors  teaches  students  how  to  become  effective  life-long  learners  and  guides  them  in  self- 
regulatory  practices  to  foster  a  sense  of  competence  and  honing  their  own  critical-thinking  skills 
(McGlynn,  2005),  as  well  as  metacognition  and  nurture  self-regulation  (Muller,  1998). 

In  summary,  this  literature  review  identified  several  key  attributes  of  21st  century  learners 
including  the  following: 

a.  desire  for  meaningful  work 

b.  are  self-directed 

c.  prefer  social  interaction  &  collaboration 

d.  have  a  sense  of  competence 

e.  have  strong  visual-spatial  skills 

f.  have  strong  parallel  processing  skills 

g.  are  technologically  literate 

This  review  also  identified  best  practices  for  21st  century  learners.  Teaching  practices  that  draw 
on  learners’  strengths  and  preferences  include: 

a.  cooperative  learning 

b.  technology  tools 

c.  visual  tools 

d.  problem-based  learning 

e.  content-related  parallel  processing 

f.  opportunities  for  choices  and  decision-making  and  selecting  learning  strategies 

g.  establishing  high  expectations 

h.  frequent  praise  and  feedback 
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i.  metacognition 

j.  meaningful  work 

The  most  effective  learning  context  is  one  in  which  the  teaching  methods  optimize  learners’ 
strengths  and  preferences. 


2.  Data  Gathering  Tools 

To  investigate  the  relationship  between  technical  training  teaching  methods  and  trainees’ 
learning  preferences,  classroom  observation  and  preference  inventories  were  employed. 

A.  Classroom  Observation  was  the  method  selected  to  assess  the  degree  of  alignment  between 
recommended  21st  century  training  practices  and  current  technical  training  practices.  Classroom 
Observations  have  been  used  in  traditional  teacher  training  for  decades.  One  such  example  is  the 
PRAXIS  III,  based  upon  nineteen  criteria  divided  into  four  domains  developed  by  the 
Educational  Testing  Service.  A  three-point  rubric  rating  system  is  used  to  score  observed 
evidence  for  each  criteria  from  which  sub-scores  are  computed  for  the  four  domains.  In  states 
using  the  PRAXIS  III,  first  year  teachers  must  achieve  a  passing  rating  in  each  of  the  domains  to 
be  licensed  to  teach  in  public  schools.  This  approach  to  classroom  observation  served  as  a  model 
for  developing  an  observation  rubric  for  this  pilot  study. 

The  21st  Century  Technical  Training  Observation  Rubric  (TTOR)  was  developed  using  the 
learner  attributes/preferences  and  best  instructional  practices  identified  in  the  review  of  literature 
(See  Attachment  A).  The  rubric  has  nine  criteria  with  three  or  more  examples  of  possible 
evidence  one  might  observe.  For  example,  possible  evidence  for  the  criteria  self-direction  are:  a) 
students  exhibiting  confidence  in  their  own  ability  to  leam  with  statements  like  “I  know  how  to 
do  this,”  getting  right  to  work  on  assignments,  helping  peers,  elaborating  beyond  basic 
understandings;  b)  students  selecting  appropriate  learning  strategies  like  drawing  a  web  or 
diagram  following  an  explanation  to  clarify  meaning  for  self;  c)  students  making  choices  about 
their  learning  experiences  by  selecting  between  options  in  assignments  or  selecting  the  sequence 
for  completing  a  set  of  activities. 

The  first  eight  criteria  describe  21st  century  instruction  and  the  ninth  criteria  permitted  the 
documentation  of  traditional  instruction: 

1.  Training  is  meaningful  to  trainees 

2.  Training  supports  self-direction 

3.  Training  incorporates  metacognition 

4.  Training  supports  collaboration 

5.  Training  supports  a  trainee’s  sense  of  competence 

6.  Training  supports  parallel  processing 

7.  Training  uses  technology  to  support  learning 

8.  Training  includes  trainees  learning  in  a  variety  of  ways 

9.  Training  uses  traditional  methodology 

Prior  to  implementation,  the  rubric  was  piloted  by  the  observation  team  using  several  teaching 
videos  to  allow  the  raters  to  apply  the  criteria,  discuss  observed  evidence  for  clarity,  and  to 
develop  inter-rater  reliability. 
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B.  The  Grasha-Reichmann  Learning  Styles  Inventory  (Attachment  B)  was  selected  as  the 
instrument  to  measure  the  learning  preferences  of  a  sample  group  of  technical  trainees.  The 
instrument  is  a  60  item  Likert-type  rating  scale  with  response  options  from  1-  strongly  disagree 
to  5  strongly  agree.  The  inventory  is  scored  by  averaging  the  ratings  for  ten  items  associated 
with  six  different  learning  styles: 

Independent  students  prefer  independent  study,  self-paced  instruction,  and  working 
alone  on  course  projects. 

Dependent  learners  look  to  the  teacher  and  to  peers  as  a  source  of  structure  and  guidance 
and  prefer  an  authority  figure  to  tell  them  what  to  do. 

Competitive  students  learn  in  order  to  perform  better  than  their  peers  do  and  to  receive 
recognition  for  their  accomplishments. 

Collaborative  learners  acquire  information  by  sharing  and  cooperating  with  the 
Instructor  and  peers.  They  prefer  lectures  with  small  group  discussions  and  group 
projects. 

Avoidant  learners  are  not  enthused  about  attending  class  or  acquiring  class  content.  They 
are  typically  uninterested  and  are  sometimes  overwhelmed  by  class  activities. 

Participant  learners  are  interested  in  class  activities  and  discussion,  and  are  eager  to  do 
as  much  class  work  as  possible.  They  are  keenly  aware  of,  and  have  a  desire  to  meet 
teacher  expectations. 

Based  on  the  literature  about  21st  century  learners,  trainees  were  expected  to  score  higher  in 
Collaborative  and  Participant  styles,  and  to  score  lower  in  Competitive,  Independent,  and 
Avoidant  styles.  A  moderate  score  in  the  Dependent  style  was  also  anticipated  based  on  the  need 
21st  century  learners  have  for  authority  figures  to  provide  high  expectations,  frequent  praise,  and 
feedback. 

While  the  six  learning  styles  identified  by  the  LSI  correlate  to  some  degree  with  the  learning 
preferences  being  studied,  they  did  not  directly  align  with  the  21st  century  preferences  identified 
in  our  literature  review  and  no  items  addressed  technology  use.  Therefore,  ten  additional  items 
related  to  technology  use  for  learning  were  added  to  the  end  of  the  LSI,  creating  a  70  item 
inventory.  Then  the  items  from  the  inventory  were  mapped  to  the  identified  21st  century 
attributes  to  enable  the  computation  of  sub-scores  representative  of  21st  century  learning 
preferences.  The  mapping  process  identified  six  LSI  items  related  to  each  21st  century 
preference,  with  the  exception  of  parallel  processing  for  which  no  related  items  could  be 
identified.  Some  items  were  negatively  related  to  a  21st  century  preference  so  those  ratings  were 
interpolated  (1=5,  2=4,  etc)  prior  to  calculating  the  sub-scores.  Table  1  displays  the  LSI  item 
mapping.  Interpolated  items  are  designated  with  a  negative  sign  (-). 


Table  1.  LSI 

Items  for 

Measuring 

21st 

Century  Preferences 

21st  C.  Preference 

LSI  Items 

Meaningful 

-8 

12 

-14 

18 

30 

36 

Self-direction 

19 

-28 

43 

48 

49 

5 

Metacognition 

7 

24 

25 

31 

53 

55 

Collaboration 

3 

9 

15 

21 

39 

45 

Competence 

4 

6 

10 

29 

46 

59 

Technology 

62 

64 

65 

66 

67 

70 

Traditional 

1 

16 

22 

34 

40 

52 

49 


Once  the  LSI  items  for  the  21st  century  preference  were  identified,  a  comparison  was  made  with 
the  items  associate  with  the  six  LSI  learning  styles  to  note  which  items  were  shared  by  each  LSI 
learning  styles  and  21st  century  preferences.  This  process  suggested  the  anticipated  LSI  styles 
scores:  high  scores  on  the  21st  century  preferences  would  also  be  reflected  in  higher  preferences 
for  the  Participant  and  Collaborative  learning  styles,  moderate  preferences  for  the  Dependent, 
Independent,  and  lower  preferences  for  the  Competitive  and  Avoidant  styles.  The  numbers  of 
items  associated  with  each  LSI  learning  style  used  to  measure  each  21st  century  preference  are 
noted  in  Table  2.  Negatively  related  items  are  noted  with  a  negative  sign  (-). 


Table  2.  Number  of  Shared  Items  Measuring  21st  Century  Preferences  and  LSI  Learning 

Styles 


21st  C.  Preference 

Meaningful 

Self-direction 

Metacognition 

Collaboration 

Competence 

Technology 


LSI  Learning  Style  Items 

4  Participant,  2(-)  Avoidant 

2  Independent^  Participant,!  Collaborative, l(-)  Dependent 
4  Independent,!  Participant,!  Competitive 

6  Collaborative 

3  Dependent^  Competitive,!  Participant 
6  Technology 


C.  The  Grasha-Reichmann  Teaching  Styles  Inventory  (Attachment  C)  was  selected  as  the 
instrument  to  measure  the  teaching  preferences  of  a  sample  group  of  technical  training 
instructors.  The  instrument  is  a  50  item,  Likert-type  rating  scale  with  response  options  from  1- 
strongly  disagree  to  5  strongly  agree.  The  inventory  is  scored  by  averaging  the  ratings  for  eight 
items  associated  with  each  teaching  styles: 

Expert  teachers  possess  knowledge  and  expertise  that  students  need;  strives  to  maintain  status  as 
an  expert  among  students  by  displaying  detailed  knowledge  and  by  challenging  students  to 
enhance  their  competence;  concerned  with  transmitting  information  and  insuring  that  students 
are  well  prepared. 

Formal  Authority  teachers  possess  status  among  students  because  of  knowledge  and  role  as 
instructor;  concerned  with  providing  positive  and  negative  feedback,  establishing  learning 
goals,  expectations,  and  rules  of  conduct  for  students;  concerned  with  the  correct,  acceptable, 
and  standard  ways  to  do  things  and  with  providing  students  with  the  structure  they  need  to  learn. 

Personal  Model  teachers  believe  in  "teaching  by  personal  example"  and  establish  a  prototype 
for  how  to  think  and  behave;  oversee,  guide,  and  direct  by  showing  how  to  do  things,  and 
encouraging  students  to  observe  and  then  to  emulate  the  instructor's  approach. 

Facilitator  teachers  emphasize  the  personal  nature  of  teacher- student  interactions;  guide  and 
direct  students  by  asking  questions,  exploring  options,  suggesting  alternatives,  and  encouraging 
them  to  develop  criteria  to  make  informed  choices;  develop  in  students  the  capacity  for 
independent  action,  initiative,  and  responsibility  while  providing  as  much  support  and 
encouragement  as  possible. 

Delegator  teachers  are  concerned  with  developing  students'  capacity  to  function  in  an 
autonomous  fashion  and  work  independently  on  projects  or  as  part  of  autonomous  teams  with  the 
teacher  available  upon  request  as  a  resource  person. 
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As  with  the  LSI,  no  items  on  the  TSI  addressed  technology  use,  so  parallel  versions  of  the  ten 
items  related  to  technology  use  developed  for  the  LSI  were  added  to  the  end  of  the  TSI,  creating 
a  60  item  inventory.  The  score  for  technology  was  computed  by  averaging  the  ratings  for  the  ten 
technology  items. 

Grasha’s  research  (1996,  2002)  identified  four  common  clusters  of  teaching  styles  and  associated 
student  learning  styles  that  resulted  in  the  most  effective  teaching-learning  dynamic.  Each  of  the 
Teaching  Styles  Clusters  are  made  up  of  two  or  three  primary  teaching  styles  and  one  or  more 
secondary  styles.  Below  is  a  brief  description  of  the  four  clusters  and  the  learning  styles  for 
which  they  are  most  effective. 

1-  Expert/Formal  Authority  with  Personal  Model/Facilitator/Delegator.  Most  effective  with 
Dependent/Participant/Competitive  learners  who  have  little  or  no  need  for  a  relationship  with  the 
instructor  or  other  students.  The  instructor  has  low  sensitivity  to  students’  learning  style 
preferences.  Instruction  includes:  exams,  grades,  guest  speakers/interviews,  lectures,  teacher- 
centered  questioning  and  discussions,  term  papers,  tutorials,  and  technology-  based  presentations 
(i. Powerpoint). 

2-  Personal  Model/Expert/Formal  Authority  with  Facilitator/Delegator.  Most  effective  with 
Participant/Dependent/Collaborative  learners  who  desire  relationships  with  instructors  and 
fellow  students.  Instructor  has  moderate  to  high  sensitivity  to  students’  learning  style 
preferences,  engenders  respect  and  is  liked  by  students.  Instruction  includes:  role  modeling, 
illustration,  demonstration,  examples,  discussions  of  alternative  approaches,  sharing  thought 
processes  for  obtaining  answers,  sharing  personal  experiences,  coaching  and  guiding  with 
feedback,  students  actively  emulate  instructor. 

3-  Facilitator/Personal  Model/Expert  w/Formal  Authority/Delegator.  Most  effective  with 
Collaborative/Participant/Independent  learners  who  are  willing  to  take  initiative  and  accept 
responsibility  for  learning  tasks.  Instructor  exercises  some  control  over  learning  processes  to 
facilitate  learning;  has  professional,  friendly  and  warm  relationships  with  students,  and  has 
moderate  to  high  sensitivity  to  students’  learning  preferences.  Instruction  includes:  case  studies, 
cognitive  mapping/visual  tools  discussions,  critical  thinking  discussions,  fishbowl  discussions, 
guided  readings,  key  statements  with  discussion,  lab  projects,  problem-based  learning,  role 
playing,  simulations,  roundtable  discussions,  and  peer  teaching. 

4-  Delegator/Facilitator/Expert  w/  Formal  Authority/Personal  Model.  Most  effective  with 
Independent/Collaborative/Participant  learners  who  take  responsibility  and  initiative.  Instructor 
gives  up  control  over  students,  develops  rapport,  empowers  students,  helps  students  develop 
good  relationships  with  peers,  has  moderate  to  high  sensitivity  to  students’  learning  preferences, 
approaches  learning  as  working  together  with  students  in  a  more  relational  based  dynamic. 
Instruction  includes:  contracts,  symposiums,  debates,  teams/trios,  independent  study/research, 
jigsaws,  self-paced  modules  of  instruction,  panel  discussions,  position  papers,  self-discovery 
activities,  small  work  groups,  journals,  and  internships/practicum. 

Based  on  the  literature  about  21st  century  learners’  attributes  and  preferences,  and  the  best 
practices  for  21st  century  learners,  the  Cluster  2  combination  of  teaching  styles  best  aligns  with 
the  identified  needs  of  21st  century  learners.  Given  the  expectation  that  current  technical  training 
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instruction  was  not  aligned  with  the  needs  of  21st  century  learners,  Cluster  1  teaching  styles 
scores  were  anticipated  for  the  sample  of  technical  training  instructors. 


3.  Data  Collection  and  Analysis 

Data  were  collected  in  October,  2008  during  three  days  of  technical  training  at  Sheppard  AFB. 
Instructors  and  students  in  six  technical  training  courses  were  observed  and  asked  to  complete 
the  preference  inventories.  Inventory  responses  were  compiled  using  electronic  scanning 
equipment  and  data  were  analyzed  using  add-in  statistical  analysis  software  for  Microsoft  Excel 
2007.  Observation  ratings  were  complied  and  analyzed  in  Excel  2007. 

A.  Observations.  Instruction  (n=17)  in  six  technical  training  courses  was  observed  to 

document  the  teaching  methods  used  in  a  range  of  career  fields.  One  Instructors’  Training 
class  was  also  observed.  The  observation  protocol  involved  teams  of  two  or  four  raters 
observing  each  class  for  20-30  minutes,  note-taking  during  the  observation  period 
followed  by  assigning  ratings  based  on  the  evidence  noted.  Raters  completed  their  ratings 
individually  for  each  criteria  using  a  0-3  rating  scale  (0=  not  observed,  1=  seldom 
observed,  2=  observed  about  half  of  the  time,  3=  observed  most  or  all  of  the  time). 
Following  each  morning  or  afternoon  observation  period,  all  four  observers  gathered  to 
discuss  their  evidence  and  their  individual  ratings  for  each  class.  Most  observations 
resulted  in  common  ratings  across  the  observers  but  when  differences  in  individual 
ratings  occurred,  discussions  of  evidence  resulted  in  the  raters  reaching  consensus.  Even 
on  the  few  occasions  when  team  teaching  was  observed,  one  rating  for  instruction  was 
recorded  for  the  class.  Ratings  for  the  nine  criteria  were  totaled  for  each  observed  class 
resulting  in  a  total  observation  score.  The  optimal  total  score  was  26,  based  on  a  score  of 
3  for  the  eight  21st  century  criteria  and  a  score  of  2  for  traditional  methodology.  Table  3 
displays  a  summary  of  the  observation  data  by  course.  Table  4  displays  the  mean  rubric 
ratings  by  criteria. 


Table  3.  Instruction  Observation  Data  Summary 


Course  (#  observed) 

Mean 

%  of  optin 

Maintenance  (2) 

15 

57.7% 

Munitions  (2) 

14 

53.8% 

Communications  (4) 

15.5 

59.6% 

HVAC  (2) 

13.5 

51.9% 

Medical  (6) 

13.3 

51.1% 

Pharmacy  (1) 

20 

76.9% 

Instructor  Training  (1) 

11 

42.3% 

All  Courses 

14.6 

56.2% 
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Table  4.  Mean  Observation  Rubric  Ratings  by  21st  Century  Criteria 


21st  C.  Criteria 

Meaningful 

Self-direction 

Metacognition 

Collaboration 

Competence 

Parallel  processing 

Technology 

Variety 

Traditional 


Mean  Rating  (0-3) 

2.00 

.94 

1.80 

1.53 

1.76 

1.47 

.11 

2.18 

2.59 


Overall,  observed  instruction  was  dominantly  traditional  using  lecture  with  slides.  These 
methods  are  used  in  Grasha’s  Teaching  Styles  Cluster  1.  The  course  with  the  highest  level  of 
observed  alignment  with  21st  century  instruction  was  Pharmacy  (76.9  percent),  in  which  students 
were  actively  participating  by  working  at  lab  stations  mixing  a  prescription  cream.  The  course 
with  the  lowest  observed  alignment  was  in  the  instructor  training  course,  where  students  sat  in 
rows  listening  to  a  lecture  with  slides. 

The  mean  criteria  ratings  document  that  the  observed  technical  training  instruction  addressed  the 
identified  21st  century  learning  preferences  less  than  half  of  the  time.  Two  criteria  received  mean 
ratings  of  2.00  and  2.18,  suggesting  that  half  of  the  time  instruction  was  meaningful  to  students 
and  included  some  variety  (auditory,  visual,  kinesthetic).  Overall,  observation  data  indicated  that 
the  majority  of  the  time  technical  training  instruction  was  not  aligned  with  21st  century  best 
practices. 

B.  The  Grasha-Reichmann  Learning  Styles  Inventory  (LSI)  was  completed  by  a  sample  of 
technical  trainees  (n=172)  to  identify  their  learning  style  preferences.  Mean  rating  scores  were 
computed  for  each  of  the  six  LSI  learning  styles  plus  technology  and  for  seven  21st  century 
learning  preferences.  The  mean  learning  styles  scores  are  presented  from  highest  to  lowest  in 
Tables  5  and  the  mean  ratings  for  the  21st  century  preferences  are  presented  from  highest  to 
lowest  in  Table  6. 

Table  5.  Mean  Ratings  (scale  1-5)  for  Technical  Trainees’  LSI  Learning  Styles 


Learning  Style  Mean 

Participant  4.13 

Dependent  3.91 

Collaborative  3.9 

Technology  3.75 

Independent  3.31 

Competition  2.71 

Avoidant  2.26 
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Table  6.  Mean  Ratings  (scale  1-5)  for  Technical  Trainees’  21st  Century  Preferences 


21st  C.  preference 

Mean 

Meaningful 

4.03 

Collaboration 

3.95 

Metacognition 

3.83 

Competence 

3.82 

Technology 

3.76 

Self-direction 

3.65 

Traditional 

3.27 

The  mean  ratings  were  highly  reflective  of  the  range  of  responses  with  medians  and  modes 
similar  to  the  mean  ratings.  The  notable  exception  was  technology,  where  the  mean  was  3.74 
and  the  median  3.8,  but  the  mode  (the  most  frequent  response)  was  5. 

These  data  suggest  that  the  sampled  technical  trainees  have  similar  learning  preferences  as  those 
identified  in  the  literature  review  for  21st  century  learners.  These  findings  support  the  hypothesis 
that  technical  trainees  are  21st  century  learners  with  21st  century  learning  preferences. 

Given  the  preferred  learning  style  composite  for  technical  trainees  was  Participant/ 
Dependent/Collaborative,  Grasha’s  (1996,  2002)  research  suggests  the  most  effective  instruction 
is  Teaching  Styles  Cluster  2-  Personal  Model/Expert/Formal  Authority  with  facilitator/delegator. 
Instruction  aligned  with  trainees’  preferences  would  include:  role  modeling,  illustration, 
demonstration,  examples,  discussions  of  alternative  approaches,  sharing  thought  processes  for 
obtaining  answers,  sharing  personal  experiences,  coaching  and  guiding  with  feedback,  trainees 
actively  emulating  the  instructor,  cooperative  learning,  visual  tools,  problem-based  learning, 
choices,  and  metacognition. 


C.  The  Grasha-Reichmann  Teaching  Styles  Inventory  was  completed  by  a  sample  of  technical 
training  instructors  (n=22)  to  identify  their  teaching  style  preferences  and  determine  the  level  of 
alignment  between  the  teaching  preferences  of  technical  training  instructors  and  identified  “best 
practices”  to  meet  the  needs  of  21st  century  learners.  Mean  scores  were  computed  for  each  of  the 
five  teaching  styles.  The  styles  were  then  placed  in  rank  order  from  highest  to  lowest  and  used  to 
identify  the  Teaching  Style  Cluster  preferred  by  the  sample  of  instructors.  The  mean  teaching 
styles  scores  are  presented  from  highest  to  lowest  in  Table  7. 

Table  7.  Instructors’  Mean  Ratings  on  TSI  Teaching  Styles 


Teaching  Style 

Personal  Model 
Expert 

Formal  Authority 
Facilitator 
Delegator 
Technology 


Mean  Rating  (scale  1-5) 

3.48 

3.44 

3.37 

3.04 

2.60 

2.57 
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Mean  ratings  indicated  that  instructors  preferred  Grasha’s  Teaching  Styles  Cluster  2-  Personal 
Model/Expert/Formal  Authority  with  facilitator/delegator.  This  Teaching  Styles  Cluster  aligns 
with  technical  trainees’  Participant/  Dependent/Collaborative  learning  styles  composite.  The 
teaching  methods  associated  with  these  preferences  include:  role  modeling,  illustration, 
demonstration,  examples,  discussions  of  alternative  approaches,  sharing  thought  processes  for 
obtaining  answers,  sharing  personal  experiences,  coaching  and  guiding  with  feedback,  and 
trainees  actively  emulating  the  instructor.  However,  observed  instruction  was  dominantly  lecture, 
which  is  representative  of  Teaching  Styles  Cluster  1.  This  misalignment  between  instructors’ 
preferences  and  classroom  observation  data  suggests  that  while  instructors  would  prefer  to  teach 
using  21st  century  best  practices  they  are  not  currently  using  these  methods  in  technical  training. 
In  addition.  Technology  received  a  2.57  mean  rating  by  instructors  and  a  3.75  mean  rating  by 
trainees,  indicating  that  while  trainees  prefer  greater  technology  use  in  training  that  do 
instructors,  both  groups  agree  that  training  should  include  the  use  of  email,  electronic  course 
materials,  PDAs,  computers,  and  other  technology  tools. 

6.  Findings  and  Recommendations 

The  review  of  literature  identified  several  common  preferences  of  21st  century  learners.  They 
desire  meaningful  work,  are  self-directed,  prefer  working  collaboratively,  have  a  sense  of 
competence,  have  strong  visual-spatial  and  parallel  processing  skills,  and  are  technologically 
intuitive.  The  Grasha-Reichmann  Learning  Styles  Inventory  (LSI)  was  administered  to  172 
trainees  to  identify  their  preferred  learning  styles  and  to  determine  whether  they  have  the  21st 
century  learning  preferences  identified  in  the  literature.  Trainees’  responses  on  the  LSI  indicate 
that  AF  technical  trainees  are  21st  century  learners  with  21st  century  strengths  and  learning 
preferences  as  reflected  in  a  learning  styles  composite  of  Participant/Dependent/Collaborative. 

The  Grasha  Teaching  Styles  Inventory  (TSI)  was  administered  to  22  technical  training 
instructors  in  six  courses  to  identify  their  preferred  Teaching  Styles  Cluster.  Instructors’ 
responses  on  the  TSI  indicated  that  they  preferred  Teaching  Style  Cluster  2-  Personal 
Model/Expert/Formal  Authority  with  secondary  preferences  for  the  facilitator/delegator  styles. 
Grasha’s  research  suggests  that  Teaching  Styles  Cluster  2  best  supports  students  with  the 
Participant/Dependent/Collaborative  learning  styles  composite.  This  finding  suggests  that 
technical  training  instructors’  preferred  teaching  cluster  was  aligned  with  the  learning 
preferences  of  21st  century  technical  trainees,  with  the  exception  of  technology  use  for  learning. 
Trainees  preferred  more  technology  use  in  training  than  did  their  instructors.  The  diagram  below 
illustrates  these  relationships. 


TSI  (instructors)  Preferences  LSI  (students)  preferences 


Personal  Model 

3.48^— 

- -7  Participant 

4.1 

Avoidant 

2.3 

Expert 

3.44  — ^ 

-  Dependent 

3.9 

Independent 

3.3 

Formal  Authority 

3.37 

-  Collaborative 

3.9 

Competitive 

2.7 

Facilitator 

3.04 

Delegator 

2.60 

Technology 

2.57  — 

Technology 

3.7 

Higher  mean  ratings  in  the  Participant  and  Collaborative  styles  were  anticipated  and  validated  by 
the  data.  Moderate  ratings  were  anticipated  for  the  Dependent  and  Independent  styles  with  low 


55 


ratings  for  the  Competitive  and  Avoidant  styles.  Trainee  data  resulted  in  these  anticipated 
ratings  with  the  exception  of  a  slightly  higher  than  anticipated  Dependent  style  score.  Even  so, 
the  anticipated  learning  style  composite  was  validated. 

An  interesting  finding  from  these  data  was  that  the  LSI  and  TSI  data  indicated  alignment 
between  instructors’  and  trainees’  preferences,  but  the  classroom  observation  ratings  using  the 
21st  Century  Technical  Training  Observation  Rubric  indicated  instruction  was  predominantly 
traditional.  Observation  ratings  and  notes  documented  the  use  of  lectures,  teacher-centered 
questioning,  and  slide  presentations.  These  methods  are  representative  of  Grasha’s  Teaching 
Styles  Cluster  1-  Expert/Formal  Authority  with  the  secondary  styles  of  personal 
model/facilitator/  delegator.  So  while  instructors  preferred  Teaching  Styles  Cluster  aligned  with 
the  preferences  of  technical  trainees,  instructors  were  not  observed  teaching  with  Cluster  2 
instruction.  In  exploring  this  contradiction,  it  is  helpful  to  understand  the  finer  differences 
between  these  two  clusters. 

Teaching  Styles  Clusters  1  and  2  share  two  primary  and  two  secondary  styles;  Expert/Formal 
Authority  and  facilitator/delegator.  Both  clusters  also  address  the  needs  of  participant  and 
dependent  learners,  but  differ  in  relationships  with  students.  Cluster  1  emphasizes  expertise  and 
authority,  and  employs  traditional  instructional  strategies  which  are  most  effective  with 
competitive  learners  who  do  not  desire  relationships  with  instructors.  Cluster  2  emphasizes  the 
personal  model  which  is  most  effective  with  collaborative  learners  who  do  desire  relationships 
with  instructors.  The  literature  on  21st  century  learners  and  the  findings  from  this  pilot  study 
suggest  that  technical  trainees  are  Collaborative  not  Competitive  in  their  learning  preferences, 
and  value  relationships.  They  are  Dependent  and  Independent  learners  in  that  they  desire 
choices,  feeling  competent,  need  feedback  and  guidance  but  not  control  by  instructors.  These  are 
important  distinctions  that  directly  impact  the  effectiveness  of  instruction. 

The  alignment  between  instructors’  and  trainees’  preferences  is  good  news,  in  that  it  suggests 
that  instructors  possess  the  desire  and  inclination  to  teach  in  ways  that  support  21st  century 
learners’  needs  for  participation,  support  and  collaboration.  However,  observed  instruction  did 
not  utilize  recommended  21st  century  practices  such  as:  cooperative  learning,  role  modeling, 
illustrations,  problem-solving,  multiple  examples,  discussions  of  alternative  approaches,  sharing 
of  thought  processes  for  obtaining  answers,  sharing  of  personal  experiences,  coaching  and 
guiding  with  feedback,  or  visual  tools.  Since  instructors  preferred  but  were  not  observed  enacting 
21st  century  teaching  methods  in  the  classroom,  there  is  need  for  further  exploration  of  this 
contradiction.  Perhaps  aspects  of  the  training  environment  or  the  structure  of  the  curriculum  are 
constraining  instructors  from  teaching  in  their  preferred  styles  cluster.  This  is  certainly  an  area 
worthy  of  further  research. 

The  findings  of  this  pilot  study  suggest  that  technical  training  instruction  could  be  better  aligned 
to  meet  both  instructors’  preferred  teaching  styles  cluster  and  trainees’  learning  preferences. 
Adjustments  in  teaching  methods  could  be  made  to  accommodate  these  preferences.  The 
incorporation  of  methods  like  cooperative  learning,  problem-based  learning,  metacognition,  and 
visual  tools  would  be  fairly  low-cost  modifications  that  could  significantly  improve  engagement 
and  learning.  The  following  is  an  example  of  using  a  cooperative  learning  approach  to  quickly 
change  the  learning  experience  observed  during  a  classroom  observation  completed  during  this 
study: 
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The  observed  teaching  method  involved  the  instructor  reading  sections  of  a  technical 
order  (TO)  and  calling  on  various  students  to  read  aloud  statements  from  the  TO.  During  class, 
several  students  were  trying  to  keep  from  nodding  off  and  others  were  told  by  the  instructor  to 
stand  to  avoid  falling  asleep.  A  21st  century  interactive,  collaborative  approach  would  use 
cooperative  learning  to  teaching  this  information.  To  set  up  the  cooperative  learning  experience, 
the  instructor  would  make  4  enlarged  copies  of  the  TO,  cut  the  statements  apart  and  place  the 
statements  for  each  copy  in  a  large  envelope.  The  students  would  be  placed  in  trios  and  each 
group  would  receive  an  envelope  with  the  TO  statements.  Given  a  limited  but  sufficient  time 
block  to  complete  their  work  student  trios  would  reconstruct  the  TO  by  reading,  discussing,  and 
putting  the  statements  in  the  proper  order.  When  finished,  they  would  tape  the  reconstructed 
document  on  a  poster  page  and  display  it  on  the  wall.  Then  the  instructor  would  reveal  the  TO  on 
a  slide  and  ask  the  teams  to  check  how  accurately  they  reconstructed  the  TO  and  discuss 
how/why  they  made  their  ordering  decisions. 

By  using  this  approach,  every  student  must  participate  in  not  just  reading  the  TO  statements,  but 
in  discussing  and  making  sense  of  the  statements  with  their  teammates  in  order  to  put  the 
statements  in  the  proper  order.  This  process  engages  students  in  collaboration  and  shared 
meaning-making  through  a  shared  purpose.  They  must  think  through  the  TO  task  process  and 
make  decisions  about  which  steps  should  be  completed  in  which  sequence.  These  are  the  kinds 
of  21st  learning  methods  that  could  quickly  be  included  in  training  at  minimal  cost.  A  “toolbox” 
of  these  kinds  of  21st  century  instructional  strategies  could  be  incorporated  into  Instructor 
Training. 

More  technology  access  and  use  in  the  classroom  is  another  avenue  to  engage  trainees  and 
improve  training  outcomes.  Trainees  expressed  the  desire  for  more  technology  use  in  the 
classroom  and  for  digital  access  to  course  materials.  Even  though  instructors’  responses  on 
technology  items  were  lower  than  those  of  trainees,  they  did  indicate  they  would  like  to 
incorporate  more  technology  into  the  training  process.  Adjustments  in  teaching  to  include  more 
technology  tools  would  accommodate  both  instructors’  and  trainees’  preferences.  A  variety  of 
technologies  beyond  the  common  power-point  lecture  are  available  to  incorporate  into  the 
learning  environment.  Electronically  available  course  materials,  email  communications,  online 
wikis  and  blogs  could  be  incorporated  to  provide  collaboration  and  interactivity.  When  guided  by 
training  objectives,  small-scale  shifts  in  technology  use  could  offer  trainees  meaningful  learning 
experiences  to  increase  engagement  and  improve  learning  outcomes.  Further  research  is  needed 
to  determine  the  impact  of  incorporating  the  use  of  simple  technology  tools  in  technical  training. 

Several  questions  worthy  of  research  arose  from  this  study,  especially  those  that  focus  on  the 
relationships  and  interconnections  between  Learners,  Instructors,  and  Curriculum. 

•  What  role  does  curriculum  design  play  in  shaping/constraining  instruction? 

•  What  would  class  lessons  look  like  if  21st  century  criteria  were  used  to  guide 
curriculum  design? 

•  What  models  of  instruction  are  included  in  instructor  training? 

•  Do  the  methods  of  instruction  in  instructor  training  support  21st  century/Cluster  2 
teaching  methods  like:  role  modeling,  illustration,  demonstration,  examples, 
discussions  of  alternative  approaches,  sharing  thought  processes  for  obtaining 
answers,  sharing  personal  experiences,  coaching  and  guiding  with  feedback,  students 
actively  emulating  the  instructor? 


57 


•  What  would  instructor  training  look  like  if  the  21st  Century  Technical  Training 
Observation  Rubric  (TTOR)  criteria  were  used  for  assessing  instruction? 

As  a  pilot  study  with  a  small  sample  of  instructors  and  courses,  further  research  to  validate  these 
findings  would  also  be  helpful  and  appropriate  next  steps.  Technical  trainees  are  21st  century 
learners  who  deserve  to  be  fully  engaged  in  their  technical  training,  and  21st  century  instructional 
practices  are  more  likely  to  help  them  succeed.  Improving  technical  training  instruction  through 
the  use  of  21st  century  best  practices  and  incorporating  more  technology  use  in  the  learning 
process  are  viable  avenues  worthy  of  further  investigation. 
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ATTACHMENT  A 

21st  Century  Technical  Training  Observation  Rubric  (TTOR) 


TECHNICAL  TRAINING  OBSERVATION  RUBRIC 

Course_ _ _  Block _ .Unit  _ _ Instructor _ Date _ 

Rate  the  frequency  of  the  training  attribute  using  the  following  Rating  Scale: 

Indicators  and  specific  examples  of  observable  behaviors  are  provided  for  each  attribute 
0=  not  observed  1=  seldom  observed  2=  observed  about  half  the  time  3=  observed  most  or  all 
of  the  time 

_ Training  is  meaningful  to  trainees  as  evidenced  by  trainees: 

_ .  Completing  assignments  that  aligns  with  lesson  goals 

_  Making  connections  between  experiences  and  lesson  content  (“This  is  like  when  I  had  to 

...) 

_ _  Generating  questions  about  lesson  content 

Actively  participating  in  the  lesson  (making  comments,  eye  contact,  listening) 

Applying  what  they  learn  to  real-life  situations 

Transferring  learning  across  lessons/units  (using  prior  knowledge,  identifying  similarities 
between  lessons) 

Solving  real  work  problems 

_ Training  supports  self  direction  as  evidenced  by  trainees: 

Exhibiting  confidence  in  their  own  ability  to  leam  (“I  know  how  to  do  this”, 

getting  right  to  work  on  assignments,  helping  peers,  elaborating  beyond 
basic  understandings) 

_ .  Selecting  appropriate  learning  strategies  (drawing  a  web  or  diagram 

following  an  explanation  to  clarify  for  self) 

_  Making  choices  about  their  learning  experiences  (options  in  assignments 

or  selecting  the  sequence  for  completing  a  set  of  activities) 

_ Training  incorporates  metacognition  as  evidence  by  trainees: 

_  Monitoring  their  own  understanding  of  concepts  (asking  questions  to  get 

clarification) 

_  Monitoring  their  own  skill  level  (checking  to  see  how  well  they  are  doing, 

asking  for  guidance) 

_  Monitoring  the  quality  of  their  work  (checking  work,  comparing  work  to 

exemplars) 

Identifying  their  own  learning  needs  (“I  don’t  understand”,  asking  for  help, 
requesting  alternative  explanations) 

_ Training  supports  collaboration  as  evidenced  by: 

_ .  Trainees  showing  respect  for  one  another  (tone,  language,  addressing 

people  by  name) 

_ .  Trainees  and  Instructor  showing  respect  for  one  another  (tone,  language, 

addressing  by  name) 
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_  Trainees  accepting  personal  differences  (not  making  fun  of  dialects  or 

language  or  abilities) 

_  Furniture  arrangements  that  encourage  interaction  (tables  and  chairs  in 

groups) 

_  Assignments/events  that  require  cooperation/teamwork  (tasks  with  2  or 

more  trainees  discussing  and  working  together) 

_  Peer  teaching  (a  trainee  explaining  content  to  another  trainee) 

Training  supports  the  trainee’s  sense  of  competence  as  evidenced  by: 

_  Assignments  that  fits  the  trainee’s  readiness  level 

_  High,  attainable  expectations  (Instructor  states  belief  in  trainees’  capacity 

to  learn) 

_  Frequent  praise  and  recognition  from  instructor  (Good  job,  well  done) 

_  Opportunities  for  the  trainee  to  engage  in  decision-making 

_  Trainees  discuss  the  basis  of  standards,  guidelines,  and  rules 

_  Instructors  accommodate  learning  differences  (teach  to  different  learning 

styles) 

Training  supports  parallel  processing  as  evidenced  by: 

_  Flexible  instructional  materials  (all  course  materials  available  to  use  as 

needed,  easy  to  read/understand,  presented  in  a  variety  of  learning 
modalities) 

_  Muti-sensory  learning  experiences  (visual,  oral,  kinesthetic) 

_  Displays  relevant  to  current  instruction 

_  Active  manipulation/exploration  of  learning  materials  by  trainees  (trainees 

get  to  handle  materials  and  make/learn  from  mistakes  not  just  observe  a 
demonstration) 

_  Trainees  using  resources  and  strategies  in  real-life  contexts  (simulations, 

labs,  flight  line) 

_  Evaluation  and  feedback  occur  while  learning  in  happening  (Instructor 

observes  and  comments  on  performance  related  to  specific 
competencies) 

_  Trainees  have  opportunities  for  task-related  multi-tasking  (simultaneous 

use  of  course  materials,  technology,  and  coaching  while  completing  a 
task) 

Training  uses  technology  to  support  learning  as  evidenced  by  trainees: 

_  Using  technology  (computers,  hard  and  soft  trainers)  during  the  training 

event 

_  Providing  and  receiving  feedback  electronically  (email,  text  messages) 

_  Using  technology  to  access  learning  materials  (course  materials  available 

online/CD/PDA) 

_  Interacting  with  experts  in  the  field  using  technology  (chat,  email,  distance 

conference) 

_  Using  technology  to  solve  problems  (computer-based  information, 

examples,  cases) 

_  Playing  interactive  web-based  games/simulations  to  learn  specific  skills 

Training  includes  trainees  learning  in  a  variety  of  ways: 

_  Orally  (answers  to  questions,  discussions,  debates) 
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_  Written  (practice  exercises,  completing  worksheets,  essays,  tests) 

_  Visual  (drawing  diagrams,  using  cognitive  organizers) 

_  Performances  (demonstrations,  presentations,  skits,  videos) 

_  Using  different  learning  or  processing  styles 

Training  uses  traditional  methodology  as  evidenced  by 

_  Delivery  of  content  through  lecture 

_  Delivery  of  content  using  visual  aids  (PowerPoint,  video  clips,  posters) 

_  Delivery  of  content  using  demonstrations  on  hard  or  soft  trainers 

_  Interactions  between  instructor  and  trainee  rather  than  between  trainees 

_  Interactions  that  are  instructor  initiated  (instructor  asks  questions) 

_  Trainees  completing  assignments  individually 

_  Reading  assignments  and  worksheets 


64 


ATTACHMENT  B 

Grasha-Reichmann  Learning  Styles  Inventory  with  Technology  Items 

LSI  &  Technology  Survey 

Course _  Years  of  Service _  Age _  Date _  Survey _ 

Respond  to  the  items  listed  below  by  using  the  following  scale. 

Use  a  rating  of  1  if  you  strongly  disagree  with  the  statement. 

Use  a  rating  of  2  if  you  moderately  disagree  with  the  statement. 

Use  a  rating  of  3  if  you  are  undecided. 

Use  a  rating  of  4  if  you  moderately  agree  with  the  statement. 

Use  a  rating  of  5  if  you  strongly  agree  with  the  statement. 

1. 1  prefer  to  work  by  myself  on  assignments  in  my  courses 

2. 1  often  daydream  during  class. 

3.  Working  with  other  students  on  class  activities  is  something  I  enjoy  doing 

4. 1  like  it  whenever  teachers  clearly  state  what  is  required  and  expected 

5.  To  do  well,  it  is  necessary  to  compete  with  other  students  for  the  teacher’s  attention. 

6. 1  do  whatever  is  asked  of  me  to  learn  the  content  in  my  classes 

7.  My  ideas  about  the  content  are  often  as  good  as  those  in  the  textbook 

8.  Classroom  activities  are  usually  boring. 

9. 1  enjoy  discussing  my  ideas  about  the  course  content  with  other  students. 

10. 1  rely  on  my  teachers  to  tell  me  what  is  important  for  me  to  learn. 

1 1.  It  is  necessary  to  compete  with  other  students  to  get  a  good  grade. 

12.  Class  sessions  typically  are  worth  attending. 

13.1  study  what  is  important  to  me  and  not  always  what  the  instructor  says  is  important. 

14. 1  very  seldom  am  excited  about  material  covered  in  a  course. 

15. 1  enjoy  hearing  what  other  students  think  about  issues  raised  in  class. 

16. 1  only  do  what  I  am  absolutely  required  to  do  in  my  courses. 

17.  In  class,  I  must  compete  with  other  students  to  get  my  ideas  across. 

18. 1  get  more  out  of  going  to  class  than  staying  at  home. 

19. 1  learn  a  lot  of  the  content  in  my  classes  on  my  own. 

20. 1  don’t  want  to  attend  most  of  my  classes. 

21.  Students  should  be  encouraged  to  share  more  of  their  ideas  with  each  other. 

22. 1  complete  assignments  exactly  the  way  my  teachers  tell  me  to  do  them. 

23.  Students  have  to  be  aggressive  to  do  well  in  courses. 

24.  It  is  my  responsibility  to  get  as  much  as  I  can  out  of  a  course 

25. 1  feel  very  confident  in  my  ability  to  learn  on  my  own 

26.  Paying  attention  during  class  sessions  is  very  difficult  for  me  to  do. 

27. 1  like  to  study  for  tests  with  other  students. 

28. 1  do  not  like  making  choices  about  what  to  study  or  how  to  do  assignments. 

29. 1  like  to  solve  problems  or  answer  questions  before  anyone  else  can. 

30.  Classroom  activities  are  interesting 

31.1  like  to  develop  my  own  ideas  about  course  content 

32. 1  have  given  up  trying  to  leam  anything  by  going  to  class 

33.  Class  sessions  make  me  feel  like  a  part  of  a  team  where  people  help  each  other  leam. 

34.  Students  should  be  more  closely  supervised  by  teachers  on  course  projects. 
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35.  To  get  ahead  in  class,  it  is  necessary  to  step  on  the  toes  of  other  students. 

36. 1  try  to  participate  as  much  as  I  can  in  all  aspects  of  a  course 

37. 1  have  my  own  ideas  about  how  classes  should  be  run. 

38. 1  study  just  hard  enough  to  get  by. 

39.  An  important  part  of  taking  courses  is  learning  to  get  along  with  other  people 

40.  My  notes  contain  almost  everything  the  teacher  said  in  class. 

41.  Being  one  of  the  best  students  in  my  classes  is  very  important  to  me. 

42. 1  do  all  course  assignments  well  whether  or  not  I  think  they  are  interesting. 

43.  If  I  like  a  topic,  I  try  to  find  out  more  about  it  on  my  own. 

44. 1  typically  cram  for  exams. 

45.  Learning  the  material  is  a  cooperative  effort  between  students  and  teachers. 

46. 1  prefer  class  sessions  that  are  highly  organized. 

47.  To  stand  out  in  my  classes,  I  complete  the  assignments  better  than  other  students. 

48. 1  typically  complete  course  assignments  before  their  deadlines. 

49. 1  like  classes  where  I  can  work  at  my  own  pace. 

50. 1  would  prefer  that  teachers  ignore  me  in  class. 

51.1  am  willing  to  help  out  other  students  when  they  do  not  understand  something. 

52.  Students  should  be  told  exactly  what  material  is  to  be  covered  on  the  exams. 

53. 1  like  to  know  how  well  other  students  are  doing  on  exams  and  course  assignments. 

54. 1  complete  required  assignments  as  well  as  those  that  are  optional. 

55.  When  I  don’t  understand  something,  I  try  to  figure  it  out  for  myself. 

56.  During  class  sessions,  I  tend  to  socialize  with  people  sitting  next  to  me. 

57. 1  enjoy  participating  in  small  group  activities  during  class. 

58. 1  like  it  when  teachers  are  well  organized  for  a  session. 

59.1  want  my  teachers  to  give  me  more  recognition  for  the  good  work  I  do. 

60.  In  my  classes,  I  often  sit  toward  the  front  of  the  room. 

61.1  would  use  a  computer  to  access  training  course  materials. 

62. 1  would  use  a  computer  to  training  submit  assignments 

63. 1  would  use  a  mobile  or  handheld  computer  device  (Personal  Digital  Assistant  or  Cell 
Phone)  to  access  training  course  materials 

64. 1  would  use  a  mobile  or  handheld  computer  device  to  submit  training  assignments 

65. 1  would  interact  with  technical  experts  in  the  field  using  online  chat  or  email 

66. 1  would  interact  with  my  instructors  using  online  chat  or  email 

67. 1  would  like  to  learn  new  technical  skills  on  the  computer 

68. 1  would  like  to  practice  technical  skills  on  the  computer 

69. 1  would  like  to  play  interactive  web-based  games  to  learn 

70. 1  would  like  to  receive  feedback  from  instructors  by  online  chat  or  email 
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ATTACHMENT  C 

Grasha-Reichmann  Teaching  Style  Inventory  with  Technology  Items 

TSI*  &  Technology  Survey 

Course _  Years  of  Service _  Age _  Date _  Survey _ 

Respond  to  the  items  listed  below  by  using  the  following  scale. 

Use  a  rating  of  1  if  you  strongly  disagree  with  the  statement. 

Use  a  rating  of  2  if  you  moderately  disagree  with  the  statement. 

Use  a  rating  of  3  if  you  are  undecided. 

Use  a  rating  of  4  if  you  moderately  agree  with  the  statement. 

Use  a  rating  of  5  if  you  strongly  agree  with  the  statement. 


1.  Facts,  concepts,  and  principles  are  the  most  important  things  that  students  should  acquire. 

2.  I  set  high  standards  for  students  in  this  class. 

3.  What  I  say  and  do  models  appropriate  ways  for  students  to  think  about  issues  in  the 
content. 

4.  My  teaching  goals  and  methods  address  a  variety  of  student  learning  styles. 

5.  Students  typically  work  on  course  projects  alone  with  little  supervision  from  me. 

6.  Sharing  my  knowledge  and  expertise  with  students  is  very  important  to  me. 

7.  I  give  students  negative  feedback  when  their  performance  is  unsatisfactory. 

8.  Students  are  encouraged  to  emulate  the  example  I  provide. 

9.  I  spend  time  consulting  with  students  on  how  to  improve  their  work  on  individual  and/or 
group  projects. 

10.  Activities  in  this  class  encourage  students  to  develop  their  own  ideas  about  content 
issues. 

11.  What  I  have  to  say  about  a  topic  is  important  for  students  to  acquire  a  broader  perspective 
on  the  issues  in  that  area. 

12.  Students  would  describe  my  standards  and  expectations  as  somewhat  strict  and  rigid. 

13.1  typically  show  students  how  and  what  to  do  in  order  to  master  course  content. 

14.  Small  group  discussions  are  employed  to  help  students  develop  their  ability  to  think 
critically. 

15.  Students  design  one  of  more  self-directed  learning  experiences. 

16. 1  want  students  to  leave  this  course  well  prepared  for  further  work  in  this  area. 

17.  It  is  my  responsibility  to  define  what  students  must  learn  and  how  they  should  learn  it. 

18.  Examples  from  my  personal  experiences  often  are  used  to  illustrate  points  about  the 
material. 

19. 1  guide  students'  work  on  course  projects  by  asking  questions,  exploring  options,  and 
suggesting  alternative  ways  to  do  things. 

20.  Developing  the  ability  of  students  to  think  and  work  independently  is  an  important  goal. 

21.  Lecturing  is  a  significant  part  of  how  I  teach  each  of  the  class  sessions. 

22. 1  provide  very  clear  guidelines  for  how  I  want  tasks  completed  in  this  course. 

23. 1  often  show  students  how  they  can  use  various  principles  and  concepts. 

24.  Course  activities  encourage  students  to  take  initiative  and  responsibility  for  their  learning. 

25.  Students  take  responsibility  for  teaching  part  of  the  class  sessions. 

26.  My  expertise  is  typically  used  to  resolve  disagreements  about  content  issues. 
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27.  This  course  has  very  specific  goals  and  objectives  that  I  want  to  accomplish. 

28.  Students  receive  frequent  verbal  and/or  written  comments  on  their  performance. 

29. 1  solicit  student  advice  about  how  and  what  to  teach  in  this  course. 

30.  Students  set  their  own  pace  for  completing  independent  and/or  group  projects. 

31.  Students  might  describe  me  as  a  "storehouse  of  knowledge"  who  dispenses  the  facts, 
principles,  and  concepts  they  need. 

32.  My  expectations  for  what  I  want  students  to  do  in  this  class  are  clearly  defined  in  the 
syllabus. 

33.  Eventually,  many  students  begin  to  think  like  me  about  course  content. 

34.  Students  can  make  choices  among  activities  in  order  to  complete  course  requirements. 

35.  My  approach  to  teaching  is  similar  to  a  manager  of  a  work  group  who  delegates  tasks  and 
responsibilities  to  subordinates. 

36.  There  is  more  material  in  this  course  than  I  have  time  available  to  cover  it. 

37.  My  standards  and  expectations  help  students  develop  the  discipline  they  need  to  learn. 

38.  Students  might  describe  me  as  a  "coach"  who  works  closely  with  someone  to  correct 
problems  in  how  they  think  and  behave. 

39. 1  give  students  a  lot  of  personal  support  and  encouragement  to  do  well  in  this  course. 

40. 1  assume  the  role  of  a  resource  person  who  is  available  to  students  whenever  they  need 
help. 

41.1  have  used/would  use  a  computer  to  provide  online  course  materials  to  students 

42. 1  have  accepted/would  accept  students’  assignments  electronically 

43. 1  have  used/would  use  a  mobile  or  handheld  computer  device  (Personal  Digital 
Assistant  or  Cell  Phone)  to  provide  course  materials  to  students. 

44. 1  have  used/would  use  a  mobile  or  handheld  computer  device  to  receive  students’ 
assignments. 

45. 1  have  interacted/would  interact  with  experts  in  the  field  using  online  chat  or  email. 

46. 1  have  interacted/would  interact  with  my  students  using  online  chat  or  email. 

47. 1  have/would  have  students  learn  new  technical  skills  on  the  computer. 

48. 1  have/would  have  students  practice  technical  skills  on  the  computer. 

49. 1  have/would  have  students  play  interactive  web-based  games  to  learn. 

50. 1  have  provided/would  provide  feedback  to  students  using  online  chat  or  email 

51.1  am  willing  to  help  out  other  students  when  they  do  not  understand  something. 

52.  Students  should  be  told  exactly  what  material  is  to  be  covered  on  the  exams. 

53. 1  like  to  know  how  well  other  students  are  doing  on  exams  and  course  assignments. 

54. 1  complete  required  assignments  as  well  as  those  that  are  optional. 

55.  When  I  don’t  understand  something,  I  try  to  figure  it  out  for  myself. 

56.  During  class  sessions,  I  tend  to  socialize  with  people  sitting  next  to  me. 

57. 1  enjoy  participating  in  small  group  activities  during  class. 

58.1  like  it  when  teachers  are  well  organized  for  a  session. 

59. 1  want  my  teachers  to  give  me  more  recognition  for  the  good  work  I  do. 

60.  In  my  classes,  I  often  sit  toward  the  front  of  the  room. 

61.1  would  use  a  computer  to  access  training  course  materials. 

62. 1  would  use  a  computer  to  training  submit  assignments 

63. 1  would  use  a  mobile  or  handheld  computer  device  (Personal  Digital  Assistant  or  Cell 
Phone)  to  access  training  course  materials 

64. 1  would  use  a  mobile  or  handheld  computer  device  to  submit  training  assignments 
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65. 1  would  interact  with  technical  experts  in  the  field  using  online  chat  or  email 

66. 1  would  interact  with  my  instructors  using  online  chat  or  email 

67. 1  would  like  to  learn  new  technical  skills  on  the  computer 

68. 1  would  like  to  practice  technical  skills  on  the  computer 

69. 1  would  like  to  play  interactive  web-based  games  to  learn 

70. 1  would  like  to  receive  feedback  from  instructors  by  online  chat  or  email 
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ACRONYMS 


AETC 

AFRL 

AFSO-21 

ASVAB 

CAMS 

IETMs 

ISD 

JAMRS 

ESI 

TACS 

TRW 

TSI 

TTMS 

TTOR 


Air  Education  and  Training  Command 

Air  Force  Research  Laboratory 

Air  Force  Smart  Operations  21st  Century 

Armed  Services  Vocational  and  Aptitude  Battery  Test 

Core  Automated  Maintenance  System  (CAMS) 

Interactive  Electronic  Technical  Manual  (IETM) 

Instructional  Systems  Development 

Joint  Advertising,  Market  Research  and  Studies 

Learning  Styles  Inventory 

Technology  for  Agile  Combat  Support 

Training  Wing 

Teaching  Styles  Inventory 

Technical  Training  Management  System 

Technical  Training  Observation  Rubric 
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